Shell pipeline q'n

rihad · 10-10-2021, 02:58 AM

Hi. This command writes a status line every 5 seconds:

Code:

$ zpool iostat -n -q 5
              capacity     operations     bandwidth    syncq_read    syncq_write   asyncq_read  asyncq_write   scrubq_read   trimq_write
pool        alloc   free   read  write   read  write   pend  activ   pend  activ   pend  activ   pend  activ   pend  activ   pend  activ
----------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
zroot        167G   473G    398    209  17.8M  5.14M      0      0      0      0      0      0      0      0      0      0      0      0
zroot        167G   473G      0     42  19.3K   281K      0      0      0      0      0      0      0      0      0      0      0      0
^C

I'm not interested in the pool name, capacity, scrub and trim operations, so I decide to use colrm to get rid of them. First the left hand side:

Code:

$ zpool iostat -n -q 5 | colrm 1 26
 operations     bandwidth    syncq_read    syncq_write   asyncq_read  asyncq_write   scrubq_read   trimq_write
 read  write   read  write   pend  activ   pend  activ   pend  activ   pend  activ   pend  activ   pend  activ
-----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
  398    209  17.8M  5.14M      0      0      0      0      0      0      0      0      0      0      0      0
^C

So far so good. Now for the right hand side:

Code:

$ zpool iostat -n -q 5 | colrm 1 26 | colrm 85

This doesn't output anything and hangs until Ctrl+C is pressed!

I should note that it doesn't matter what the last command is, even cat -u, even head -n1 - it will hang. But if the "feeder" is killed, the expected output is delivered:

Code:

$ timeout 15 zpool iostat -n -q 5 | colrm 1 26 | colrm 85
 operations     bandwidth    syncq_read    syncq_write   asyncq_read  asyncq_write  
 read  write   read  write   pend  activ   pend  activ   pend  activ   pend  activ  
-----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  
  398    209  17.8M  5.14M      0      0      0      0      0      0      0      0  
    0     33      0   214K      0      0      0      0      0      0      0      0  
    0      0      0      0      0      0      0      0      0      0      0      0

What the $#$% is going on? Thanks for any advice.

P.S.: this doesn't hang, so it must be something in colrm (and cut -b):

Code:

$ zpool iostat -n -q 5 | cat | cat

Ser Olmy · 10-10-2021, 03:32 AM

This is a very common problem. The output from the command is being buffered. If you wait for a while, a sudden rush of output will appear when the buffer is full (which is clearly not what you want).

Try running the zpool command via stdbuf or unbuffer, like this:

Code:

stdbuf --output=L zpool iostat -n -q 5 | colrm 1 26

unbuffer -p zpool iostat -n -q 5 | colrm 1 26

I'm sure you can find other ways of doing it by Googling something like "forcing line buffering in pipes".

rihad · 10-10-2021, 04:07 AM

Quote:

Originally Posted by Ser Olmy

This is a very common problem. The output from the command is being buffered. If you wait for a while, a sudden rush of output will appear when the buffer is full (which is clearly not what you want).

Try running the zpool command via stdbuf or unbuffer, like this:

Code:

stdbuf --output=L zpool iostat -n -q 5 | colrm 1 26

unbuffer -p zpool iostat -n -q 5 | colrm 1 26

I'm sure you can find other ways of doing it by Googling something like "forcing line buffering in pipes".

Damn! You're right! Thanks a lot! It was colrm doing the buffering. This helped:

zpool iostat -n -q 5 | stdbuf -o L colrm 1 26 | colrm 85

Code:

$ zpool iostat -n -q 5 | stdbuf -o L colrm 1 26 | colrm 85
 operations     bandwidth    syncq_read    syncq_write   asyncq_read  asyncq_write  
 read  write   read  write   pend  activ   pend  activ   pend  activ   pend  activ  
-----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  
  396    208  17.7M  5.12M      0      0      0      0      0      0      0      0  
    0      0      0      0      0      0      0      0      0      0      0      0  
^C