LinuxQuestions.org - stderr and piping (not as simple as 2>&1

- Programming (https://www.linuxquestions.org/questions/programming-9/)

- - stderr and piping (not as simple as 2>&1 - so what do I do?) (https://www.linuxquestions.org/questions/programming-9/stderr-and-piping-not-as-simple-as-2-and-1-so-what-do-i-do-754214/)

stderr and piping (not as simple as 2>&1 - so what do I do?)

OK, so I'm trying to clean up the output of my ZFS backup script a bit by piping the output through perl. Here's a rough example of what I've got:

Code:

... muchos code here ...

($LAST = name of the last snapshot, $CURR = the current. The rest speaks for itself if you know ZFS - if not,

doesn't really matter, except that it transfers data via stdout/stdin.)



zfs send -R -I $LAST tank@$CURR | zfs recv -Fvd slave 2>&1 | perl -ne 'if (/receiving \w+ stream of (.*?) into/) {

    $dataset = $1; $dataset =~ s/@.*$//g;

    print ">>> Backing up $dataset\n";

}

elsif (/received 312B stream/) {

    print " no changes\n";

}

elsif (/received \w+ stream in/) {

    s/^/  /; print;

}



### Here's the problem:

elsif (/WARNING: could not send ([^@]+)/) {

#    print ">>> Skipping $1\n";

}

else {

print;

}'

So, the first three cases prettify the output; the last case does nothing... because zfs send (the *first* command in the pipe) sends it to stderr. I can't use 2>&1 or it becomes redirected into the backup stream, corrupting it.

Is there a simple way to delete that "WARNING:", or replace it with another message? Thanks in advance.

(BTW, the reason I want to replace the warning is that I do a pool-wide snapshot, then delete the ones for /var/tmp, /var/crash etc., and then "send" the entire pool; thus it warns me that the snapshots have been deleted, when in reality, that's exactly what I wanted. This seems to be the easiest way to deal with pool-wide backups and excludes, so what the heck.)

Here's what I get right now:

Code:

>> Importing pool

>> Taking snapshots

>> Cleaning out excluded snapshots

>> Starting backup...

WARNING: could not send tank/tmp@backup-20090910-2007: does not exist

>>> Backing up tank

 no changes

>>> Backing up tank/media

 no changes

>>> Backing up tank/var

WARNING: could not send tank/var/tmp@backup-20090910-2007: does not exist

  received 138KB stream in 1 seconds (138KB/sec)

>>> Backing up tank/var/log

WARNING: could not send tank/var/crash@backup-20090910-2007: does not exist

  received 194KB stream in 1 seconds (194KB/sec)

>>> Backing up tank/root

WARNING: could not send tank/usr/obj@backup-20090910-2007: does not exist

WARNING: could not send tank/usr/ports/distfiles@backup-20090910-2007: does not exist

  received 144KB stream in 1 seconds (144KB/sec)

>>> Backing up tank/usr

received 42.1KB stream in 1 seconds (42.1KB/sec)

>>> Backing up tank/usr/src_r196905

 no changes

>>> Backing up tank/usr/ports

 no changes

>>> Backing up tank/export

 no changes

>>> Backing up tank/usr/src

 no changes

>> Exporting pool

To save someone else puzzling over this, although the tags include bash and not PERL, the question is about the PERL script embedded in the bash script.

Quote:

Originally Posted by catkin (Post 3677642)

To save someone else puzzling over this, although the tags include bash and not PERL, the question is about the PERL script embedded in the bash script.

Well, not really. The problem is in the shell piping, not in the script processing it - that is mostly irrelevant.

Sorry, my mistake. How about

Code:

zfs send -R -I $LAST tank@$CURR 2>/dev/null | zfs recv -Fvd slave 2>&1 | ...

Quote:

Originally Posted by catkin (Post 3677710)

Sorry, my mistake. How about

Code:

zfs send -R -I $LAST tank@$CURR 2>/dev/null | zfs recv -Fvd slave 2>&1 | ...

Hmm yes, could work, thanks. However... that'd also remove actual errors (ones I very much want to see). What I'd prefer is to be able to "grep -v" stderr or something to that effect, before it's printed and the stream piped on to zfs recv.

OK. I worried about that after posting :(

Why can't you deal with it in the PERL?

Quote:

Originally Posted by catkin (Post 3677754)

OK. I worried about that after posting :(

Why can't you deal with it in the PERL?

The output never gets to the (or a) script; that's the problem I'm trying to solve (one way or another). :)

Gah! I feel like I'm being obtuse but I think I've finally got the requirement ...

The first command in the pipe can produce several messages on stderr but one (WARNING: could not send ...) is not an error message in this situation. You're OK that the others go to stderr but when this one appears you want to stop it going the same place as the others and instead to write an informative message to the log file.

Is that right?

Quote:

Originally Posted by catkin (Post 3677803)

Yeah, except that after running it a few times I've decided to rid the warning one altogether, rather than rewrite it (it shows up out of order and I already know what it says every time, so it's useless). Shouldn't really make a difference in the solution - although it might be possible to redirect it to /dev/null somehow.

Mmm ... assuming there's a lot of data passing through stdout on the pipeline the obvious solution of filtering through sed or awk is not appealing. How about filtering stderr from the first command in the pipeline through a co-process? Are you OK with that general idea?