LinuxQuestions.org
Visit the LQ Articles and Editorials section
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 06-04-2008, 07:03 AM   #1
lostjohnny
Member
 
Registered: May 2006
Distribution: Fedora Core 6
Posts: 37

Rep: Reputation: 15
Trying to understand pipes - Can't pipe output from tail -f to grep then grep again


Hi,

I have a log file, /tmp/GizzyLogs/2008-06-04 which is open and being written to.

I can pipe the output of 'tail /tmp/GizzyLogs/2008-06-04 ' to 'grep 134' and the output of this to 'grep SDSOffice'. As expected, the lines that contain both '134' and 'SDSOffice' are output.

Code:
[root@www ~]# tail /tmp/GizzyLogs/2008-06-04 | grep 134 | grep SDSOffice
134 2008-06-04T12:25:39.173+01:00 www gse: DISCONNECTING FROM CAMERA Site: SDSOfficeTest Camera: Dome User: GrahamTest
[etc...]
I want the output to continue as the file grows, so I try

Code:
[root@www ~]# tail -f /tmp/GizzyLogs/2008-06-04 | grep 134 | grep SDSOffice
This outputs nothing and just waits until I CTRL-C (CTRL-D has no effect)

I can pipe the output of tail to just one instance of grep, and this works fine.

Code:
[root@www ~]# tail -f /tmp/GizzyLogs/2008-06-04 | grep 134
134 2008-06-04T12:25:39.173+01:00 www gse: DISCONNECTING FROM CAMERA Site: SDSOfficeTest Camera: Dome User: GrahamTest
[etc...]
So why can't I pipe from 'tail -f' to grep to grep?

TIA,

Lost Johnny
 
Old 06-04-2008, 07:40 AM   #2
billymayday
Guru
 
Registered: Mar 2006
Location: Sydney, Australia
Distribution: Fedora, CentOS, OpenSuse, Slack, Gentoo, Debian, Arch, PCBSD
Posts: 6,678

Rep: Reputation: 122Reputation: 122
Are you sure that it works with one grep? tail -f seems to stay live while the file is open (I tried with a log file, and tail -f just outputs added lines as they are written in addition to the original tail). In other words, tail -f doesn't seem to act as an incremental command which is what you seem to me after.

Perhaps it does work that way if the file is actually closed - I won't claim to follow the man page on this point too well.

Cheers
 
Old 06-04-2008, 07:49 AM   #3
indeliblestamp
Member
 
Registered: Feb 2006
Distribution: Fedora
Posts: 341
Blog Entries: 3

Rep: Reputation: 38
I tried this too, tail -f with a single grep works but stays alive. I think that's what he wants, because when I run it in the background and add more lines to the file, grep catches it and displays it.
lostjohnny, I don't know why one pipe works but two doesn't, but have you considered grepping both your words in a single grep command using a regex? This works for me:
Code:
tail -f temp.txt | grep '134.*SDSOffice'
If you are expecting the order to be interchanged, i.e. SDSOffice...134, then it looks like you'll have to use egrep:
Code:
tail -f temp.txt | egrep '134.*SDSOffice|SDSOffice.*134'
source
 
Old 06-04-2008, 08:46 AM   #4
lostjohnny
Member
 
Registered: May 2006
Distribution: Fedora Core 6
Posts: 37

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by billymayday View Post
Are you sure that it works with one grep?
Yep. I've tried it again to be sure, but the output in my original message is copied and pasted from my terminal emulator.

Quote:
tail -f seems to stay live while the file is open (I tried with a log file, and tail -f just outputs added lines as they are written in addition to the original tail). In other words, tail -f doesn't seem to act as an incremental command which is what you seem to me after.

Perhaps it does work that way if the file is actually closed - I won't claim to follow the man page on this point too well.
I want to "follow" the file - i.e. see the new output as it appears, but I want to filter the results. I use CTRL-C when I'm finished.

What's an "incremental command", please?

Lost Johnny
 
Old 06-04-2008, 08:53 AM   #5
Agrouf
Senior Member
 
Registered: Sep 2005
Location: France
Distribution: LFS
Posts: 1,591

Rep: Reputation: 79
2 pipes should work.
Either you are unlucky and the message didn't appear when you did 2 pipes, or the second grep (SDSOffice) is wrong.
Check that you didn't put any unprintable character in the second grep.
 
Old 06-04-2008, 09:02 AM   #6
lostjohnny
Member
 
Registered: May 2006
Distribution: Fedora Core 6
Posts: 37

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by arungoodboy View Post
I tried this too, tail -f with a single grep works but stays alive. I think that's what he wants, because when I run it in the background and add more lines to the file, grep catches it and displays it.
lostjohnny, I don't know why one pipe works but two doesn't, but have you considered grepping both your words in a single grep command using a regex? This works for me:
Code:
tail -f temp.txt | grep '134.*SDSOffice'
If you are expecting the order to be interchanged, i.e. SDSOffice...134, then it looks like you'll have to use egrep:
Code:
tail -f temp.txt | egrep '134.*SDSOffice|SDSOffice.*134'
source
Thanks for the solution to my immediate problem, arungoodboy. However, I'd still really like to understand how grep can sometimes (or "often" or "usually" ) pipe to grep, but not in this case. It contradicts my understanding of pipes, which may be simplistic, but aren't pipes simple things?

I thought there'd be a way to do it in one grep statement - I've used the OR operator before (which just happens to be the same as the pipe symbol and needs to be escaped - nothing to do with this problem). I did a quick search for easy-to-read resources that would tell me how to AND, but it was lunch-time so I thought I'd come back to that after lunch. So you've saved me the effort of finding out how to effectively do an AND operation, which could have turned out to be quite substantial - don't know how long it would have taken me to think of using wildcards and realise it was the only/best way to do it. Thanks for that.

Lost Johhny

Last edited by lostjohnny; 06-04-2008 at 09:11 AM.
 
Old 06-04-2008, 09:08 AM   #7
lostjohnny
Member
 
Registered: May 2006
Distribution: Fedora Core 6
Posts: 37

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by Agrouf View Post
2 pipes should work.
Either you are unlucky and the message didn't appear when you did 2 pipes, or the second grep (SDSOffice) is wrong.
Check that you didn't put any unprintable character in the second grep.
I don't think there's anything wrong with my second grep. I tried the statement with boths greps first - no output. Then used the command history to repeat the command, but deleted the first grep - output as required:

Code:
[root@www ~]# tail -f /tmp/GizzyLogs/2008-06-04 | grep 134 | grep SDSOffice

[root@www ~]# tail -f /tmp/GizzyLogs/2008-06-04 | grep SDSOffice
134 2008-06-04T14:38:42.902+01:00 www gse: DISCONNECTING FROM CAMERA Site: SDSOfficeTest Camera: Outside User: GrahamTest
 
Old 06-04-2008, 09:15 AM   #8
Agrouf
Senior Member
 
Registered: Sep 2005
Location: France
Distribution: LFS
Posts: 1,591

Rep: Reputation: 79
That must be a buffer issue or something. I never saw anything like that, although I use double pipes quite often. If you find what it is, please tell me.
 
Old 06-04-2008, 01:50 PM   #9
indeliblestamp
Member
 
Registered: Feb 2006
Distribution: Fedora
Posts: 341
Blog Entries: 3

Rep: Reputation: 38
There's nothing a little googling can't solve! The root cause is grep's behaviour, not the pipes'. Apparently grep buffers the results so you won't (or may not) get the output instantaneously. The argument --line-buffered to the first grep solves your problem.
Code:
arun ~ $ tail -f temp.txt | grep --line-buffered 134 | grep SDSOffice
134 SDSOfficeTest blah
134 asdasdf SDSOfficeTest
Here's where I got the solution.
 
Old 06-04-2008, 03:35 PM   #10
billymayday
Guru
 
Registered: Mar 2006
Location: Sydney, Australia
Distribution: Fedora, CentOS, OpenSuse, Slack, Gentoo, Debian, Arch, PCBSD
Posts: 6,678

Rep: Reputation: 122Reputation: 122
Just to clarify my posting - when I do

tail -f /var/log/maillog

I don't return to the command line prompt, hence my suggestion that it isn't actually anything to do with grep
 
Old 06-04-2008, 05:56 PM   #11
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,226

Rep: Reputation: 2023Reputation: 2023Reputation: 2023Reputation: 2023Reputation: 2023Reputation: 2023Reputation: 2023Reputation: 2023Reputation: 2023Reputation: 2023Reputation: 2023
I've used double pipes successfully. As mentioned, its a buffering issue. In fact, if the logger is busy/frequent, you won't notice the buffering issue ie you won't need the --line-buffered (or you can just be patient ).
 
Old 06-05-2008, 12:28 AM   #12
Agrouf
Senior Member
 
Registered: Sep 2005
Location: France
Distribution: LFS
Posts: 1,591

Rep: Reputation: 79
Quote:
Originally Posted by billymayday View Post
Just to clarify my posting - when I do

tail -f /var/log/maillog

I don't return to the command line prompt, hence my suggestion that it isn't actually anything to do with grep
This is normal behaviour I believe. tail -f doesn't return and shouldn't. If it returns, this is not normal.
 
Old 06-05-2008, 12:51 AM   #13
billymayday
Guru
 
Registered: Mar 2006
Location: Sydney, Australia
Distribution: Fedora, CentOS, OpenSuse, Slack, Gentoo, Debian, Arch, PCBSD
Posts: 6,678

Rep: Reputation: 122Reputation: 122
I think that's what I was trying to get to. This in turn would suggest that it's not a grep issue at all correct?
 
Old 06-05-2008, 04:34 AM   #14
lostjohnny
Member
 
Registered: May 2006
Distribution: Fedora Core 6
Posts: 37

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by arungoodboy View Post
There's nothing a little googling can't solve! The root cause is grep's behaviour, not the pipes'. Apparently grep buffers the results so you won't (or may not) get the output instantaneously. The argument --line-buffered to the first grep solves your problem.
Code:
arun ~ $ tail -f temp.txt | grep --line-buffered 134 | grep SDSOffice
134 SDSOfficeTest blah
134 asdasdf SDSOfficeTest
Here's where I got the solution.
Thank you!

This makes sense to me now.

I'm still not entirely sure why grep (the first one) would buffer when its output is piped to another grep and not when the output is stdout, but I'm prepared to accept for now that it just does. Maybe it's something to do with how grep (the second one in this case) gets its input. It doesn't challenge my fundamental understanding of pipes, anyway.

Thanks again.

Lost Johhny
 
Old 06-05-2008, 06:42 PM   #15
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,226

Rep: Reputation: 2023Reputation: 2023Reputation: 2023Reputation: 2023Reputation: 2023Reputation: 2023Reputation: 2023Reputation: 2023Reputation: 2023Reputation: 2023Reputation: 2023
Both greps buffer (in fact all cmds do, seaprately), so the more cmds you pipe together, the bigger the effective buffer. Of course they flush when the cmd completes (although tail -f doesn't 'complete', as mentioned above).
 
  


Reply

Tags
buffer accumulation, buffer flush, stdbuf, tail grep buffer


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] tail and grep issue Wim Sturkenboom Linux - General 6 02-06-2008 07:19 AM
how to use grep and tail to get portion of a file laurasong Linux - General 7 08-19-2007 11:59 PM
grep output on stdout and grep output to file don't match xnomad Linux - General 3 01-13-2007 04:56 AM
Weird tail + grep bug Aeiri Linux - Software 2 12-15-2006 05:30 AM
Help pls!!! grep or tail->restart poj Linux - Software 3 11-26-2004 12:50 AM


All times are GMT -5. The time now is 01:29 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration