LinuxQuestions.org
Visit the LQ Articles and Editorials section
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 09-10-2009, 02:02 PM   #1
exscape
Member
 
Registered: Aug 2007
Location: Sweden
Distribution: OS X, Gentoo, FreeBSD
Posts: 82

Rep: Reputation: 15
stderr and piping (not as simple as 2>&1 - so what do I do?)


OK, so I'm trying to clean up the output of my ZFS backup script a bit by piping the output through perl. Here's a rough example of what I've got:

Code:
... muchos code here ...
($LAST = name of the last snapshot, $CURR = the current. The rest speaks for itself if you know ZFS - if not,
doesn't really matter, except that it transfers data via stdout/stdin.)

zfs send -R -I $LAST tank@$CURR | zfs recv -Fvd slave 2>&1 | perl -ne 'if (/receiving \w+ stream of (.*?) into/) {
    $dataset = $1; $dataset =~ s/@.*$//g;
    print ">>> Backing up $dataset\n";
}
elsif (/received 312B stream/) {
    print " no changes\n";
}
elsif (/received \w+ stream in/) {
    s/^/  /; print;
}

### Here's the problem:
elsif (/WARNING: could not send ([^@]+)/) {
#    print ">>> Skipping $1\n";
}
else {
print;
}'
So, the first three cases prettify the output; the last case does nothing... because zfs send (the *first* command in the pipe) sends it to stderr. I can't use 2>&1 or it becomes redirected into the backup stream, corrupting it.

Is there a simple way to delete that "WARNING:", or replace it with another message? Thanks in advance.

(BTW, the reason I want to replace the warning is that I do a pool-wide snapshot, then delete the ones for /var/tmp, /var/crash etc., and then "send" the entire pool; thus it warns me that the snapshots have been deleted, when in reality, that's exactly what I wanted. This seems to be the easiest way to deal with pool-wide backups and excludes, so what the heck.)

Here's what I get right now:
Code:
>> Importing pool
>> Taking snapshots
>> Cleaning out excluded snapshots
>> Starting backup...
WARNING: could not send tank/tmp@backup-20090910-2007: does not exist
>>> Backing up tank
 no changes
>>> Backing up tank/media
 no changes
>>> Backing up tank/var
WARNING: could not send tank/var/tmp@backup-20090910-2007: does not exist
  received 138KB stream in 1 seconds (138KB/sec)
>>> Backing up tank/var/log
WARNING: could not send tank/var/crash@backup-20090910-2007: does not exist
  received 194KB stream in 1 seconds (194KB/sec)
>>> Backing up tank/root
WARNING: could not send tank/usr/obj@backup-20090910-2007: does not exist
WARNING: could not send tank/usr/ports/distfiles@backup-20090910-2007: does not exist
  received 144KB stream in 1 seconds (144KB/sec)
>>> Backing up tank/usr
received 42.1KB stream in 1 seconds (42.1KB/sec)
>>> Backing up tank/usr/src_r196905
 no changes
>>> Backing up tank/usr/ports
 no changes
>>> Backing up tank/export
 no changes
>>> Backing up tank/usr/src
 no changes
>> Exporting pool

Last edited by exscape; 09-10-2009 at 02:07 PM.
 
Old 09-10-2009, 02:10 PM   #2
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,557
Blog Entries: 28

Rep: Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178
To save someone else puzzling over this, although the tags include bash and not PERL, the question is about the PERL script embedded in the bash script.
 
Old 09-10-2009, 02:38 PM   #3
exscape
Member
 
Registered: Aug 2007
Location: Sweden
Distribution: OS X, Gentoo, FreeBSD
Posts: 82

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by catkin View Post
To save someone else puzzling over this, although the tags include bash and not PERL, the question is about the PERL script embedded in the bash script.
Well, not really. The problem is in the shell piping, not in the script processing it - that is mostly irrelevant.
 
Old 09-10-2009, 03:04 PM   #4
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,557
Blog Entries: 28

Rep: Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178
Sorry, my mistake. How about
Code:
zfs send -R -I $LAST tank@$CURR 2>/dev/null | zfs recv -Fvd slave 2>&1 | ...
 
Old 09-10-2009, 03:26 PM   #5
exscape
Member
 
Registered: Aug 2007
Location: Sweden
Distribution: OS X, Gentoo, FreeBSD
Posts: 82

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by catkin View Post
Sorry, my mistake. How about
Code:
zfs send -R -I $LAST tank@$CURR 2>/dev/null | zfs recv -Fvd slave 2>&1 | ...
Hmm yes, could work, thanks. However... that'd also remove actual errors (ones I very much want to see). What I'd prefer is to be able to "grep -v" stderr or something to that effect, before it's printed and the stream piped on to zfs recv.
 
Old 09-10-2009, 03:40 PM   #6
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,557
Blog Entries: 28

Rep: Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178
OK. I worried about that after posting

Why can't you deal with it in the PERL?
 
Old 09-10-2009, 03:52 PM   #7
exscape
Member
 
Registered: Aug 2007
Location: Sweden
Distribution: OS X, Gentoo, FreeBSD
Posts: 82

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by catkin View Post
OK. I worried about that after posting

Why can't you deal with it in the PERL?
The output never gets to the (or a) script; that's the problem I'm trying to solve (one way or another).
 
Old 09-10-2009, 04:13 PM   #8
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,557
Blog Entries: 28

Rep: Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178
Gah! I feel like I'm being obtuse but I think I've finally got the requirement ...

The first command in the pipe can produce several messages on stderr but one (WARNING: could not send ...) is not an error message in this situation. You're OK that the others go to stderr but when this one appears you want to stop it going the same place as the others and instead to write an informative message to the log file.

Is that right?
 
Old 09-11-2009, 02:17 AM   #9
exscape
Member
 
Registered: Aug 2007
Location: Sweden
Distribution: OS X, Gentoo, FreeBSD
Posts: 82

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by catkin View Post
Gah! I feel like I'm being obtuse but I think I've finally got the requirement ...

The first command in the pipe can produce several messages on stderr but one (WARNING: could not send ...) is not an error message in this situation. You're OK that the others go to stderr but when this one appears you want to stop it going the same place as the others and instead to write an informative message to the log file.

Is that right?
Yeah, except that after running it a few times I've decided to rid the warning one altogether, rather than rewrite it (it shows up out of order and I already know what it says every time, so it's useless). Shouldn't really make a difference in the solution - although it might be possible to redirect it to /dev/null somehow.
 
Old 09-11-2009, 04:08 AM   #10
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,557
Blog Entries: 28

Rep: Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178Reputation: 1178
Mmm ... assuming there's a lot of data passing through stdout on the pipeline the obvious solution of filtering through sed or awk is not appealing. How about filtering stderr from the first command in the pipeline through a co-process? Are you OK with that general idea?
 
  


Reply

Tags
bash, pipe, piping, stderr


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
sed & piping (solved) bpwoods Linux - General 1 08-19-2009 04:36 PM
Perl stderr&stdout redirect question Fredde87 Programming 5 03-26-2009 05:43 AM
shell script: how to send stdout one place & stderr another? BrianK Programming 8 09-21-2007 07:57 AM
Shell script - stdout & stderr to terminal and file jantman Linux - Software 1 12-07-2006 05:34 PM
simple bash piping problem nadroj Linux - General 3 09-10-2005 04:12 PM


All times are GMT -5. The time now is 02:38 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration