Trying to understand pipes - Can't pipe output from tail -f to grep then grep again

lostjohnny · 06-04-2008, 07:03 AM

Hi,

I have a log file, /tmp/GizzyLogs/2008-06-04 which is open and being written to.

I can pipe the output of 'tail /tmp/GizzyLogs/2008-06-04 ' to 'grep 134' and the output of this to 'grep SDSOffice'. As expected, the lines that contain both '134' and 'SDSOffice' are output.

Code:

[root@www ~]# tail /tmp/GizzyLogs/2008-06-04 | grep 134 | grep SDSOffice
134 2008-06-04T12:25:39.173+01:00 www gse: DISCONNECTING FROM CAMERA Site: SDSOfficeTest Camera: Dome User: GrahamTest
[etc...]

I want the output to continue as the file grows, so I try

Code:

[root@www ~]# tail -f /tmp/GizzyLogs/2008-06-04 | grep 134 | grep SDSOffice

This outputs nothing and just waits until I CTRL-C (CTRL-D has no effect)

I can pipe the output of tail to just one instance of grep, and this works fine.

Code:

[root@www ~]# tail -f /tmp/GizzyLogs/2008-06-04 | grep 134
134 2008-06-04T12:25:39.173+01:00 www gse: DISCONNECTING FROM CAMERA Site: SDSOfficeTest Camera: Dome User: GrahamTest
[etc...]

So why can't I pipe from 'tail -f' to grep to grep?

TIA,

Lost Johnny

billymayday · 06-04-2008, 07:40 AM

Are you sure that it works with one grep? tail -f seems to stay live while the file is open (I tried with a log file, and tail -f just outputs added lines as they are written in addition to the original tail). In other words, tail -f doesn't seem to act as an incremental command which is what you seem to me after.

Perhaps it does work that way if the file is actually closed - I won't claim to follow the man page on this point too well.

Cheers

indeliblestamp · 06-04-2008, 07:49 AM

I tried this too, tail -f with a single grep works but stays alive. I think that's what he wants, because when I run it in the background and add more lines to the file, grep catches it and displays it.
lostjohnny, I don't know why one pipe works but two doesn't, but have you considered grepping both your words in a single grep command using a regex? This works for me:

Code:

tail -f temp.txt | grep '134.*SDSOffice'

If you are expecting the order to be interchanged, i.e. SDSOffice...134, then it looks like you'll have to use egrep:

Code:

tail -f temp.txt | egrep '134.*SDSOffice|SDSOffice.*134'

source

lostjohnny · 06-04-2008, 08:46 AM

Quote:

Originally Posted by billymayday

Are you sure that it works with one grep?

Yep. I've tried it again to be sure, but the output in my original message is copied and pasted from my terminal emulator.

Quote:

tail -f seems to stay live while the file is open (I tried with a log file, and tail -f just outputs added lines as they are written in addition to the original tail). In other words, tail -f doesn't seem to act as an incremental command which is what you seem to me after.

Perhaps it does work that way if the file is actually closed - I won't claim to follow the man page on this point too well.

I want to "follow" the file - i.e. see the new output as it appears, but I want to filter the results. I use CTRL-C when I'm finished.

What's an "incremental command", please?

Lost Johnny

Agrouf · 06-04-2008, 08:53 AM

2 pipes should work.
Either you are unlucky and the message didn't appear when you did 2 pipes, or the second grep (SDSOffice) is wrong.
Check that you didn't put any unprintable character in the second grep.

lostjohnny · 06-04-2008, 09:02 AM

Quote:

Originally Posted by arungoodboy

I tried this too, tail -f with a single grep works but stays alive. I think that's what he wants, because when I run it in the background and add more lines to the file, grep catches it and displays it.
lostjohnny, I don't know why one pipe works but two doesn't, but have you considered grepping both your words in a single grep command using a regex? This works for me:

Code:

tail -f temp.txt | grep '134.*SDSOffice'

If you are expecting the order to be interchanged, i.e. SDSOffice...134, then it looks like you'll have to use egrep:

Code:

tail -f temp.txt | egrep '134.*SDSOffice|SDSOffice.*134'

source

Thanks for the solution to my immediate problem, arungoodboy. However, I'd still really like to understand how grep can sometimes (or "often" or "usually" ) pipe to grep, but not in this case. It contradicts my understanding of pipes, which may be simplistic, but aren't pipes simple things?

I thought there'd be a way to do it in one grep statement - I've used the OR operator before (which just happens to be the same as the pipe symbol and needs to be escaped - nothing to do with this problem). I did a quick search for easy-to-read resources that would tell me how to AND, but it was lunch-time so I thought I'd come back to that after lunch. So you've saved me the effort of finding out how to effectively do an AND operation, which could have turned out to be quite substantial - don't know how long it would have taken me to think of using wildcards and realise it was the only/best way to do it. Thanks for that.

Lost Johhny

lostjohnny · 06-04-2008, 09:08 AM

Quote:

Originally Posted by Agrouf

2 pipes should work.
Either you are unlucky and the message didn't appear when you did 2 pipes, or the second grep (SDSOffice) is wrong.
Check that you didn't put any unprintable character in the second grep.

I don't think there's anything wrong with my second grep. I tried the statement with boths greps first - no output. Then used the command history to repeat the command, but deleted the first grep - output as required:

Code:

[root@www ~]# tail -f /tmp/GizzyLogs/2008-06-04 | grep 134 | grep SDSOffice

[root@www ~]# tail -f /tmp/GizzyLogs/2008-06-04 | grep SDSOffice
134 2008-06-04T14:38:42.902+01:00 www gse: DISCONNECTING FROM CAMERA Site: SDSOfficeTest Camera: Outside User: GrahamTest

Agrouf · 06-04-2008, 09:15 AM

That must be a buffer issue or something. I never saw anything like that, although I use double pipes quite often. If you find what it is, please tell me.

indeliblestamp · 06-04-2008, 01:50 PM

There's nothing a little googling can't solve! The root cause is grep's behaviour, not the pipes'. Apparently grep buffers the results so you won't (or may not) get the output instantaneously. The argument --line-buffered to the first grep solves your problem.

Code:

arun ~ $ tail -f temp.txt | grep --line-buffered 134 | grep SDSOffice
134 SDSOfficeTest blah
134 asdasdf SDSOfficeTest

Here's where I got the solution.

billymayday · 06-04-2008, 03:35 PM

Just to clarify my posting - when I do

tail -f /var/log/maillog

I don't return to the command line prompt, hence my suggestion that it isn't actually anything to do with grep

chrism01 · 06-04-2008, 05:56 PM

I've used double pipes successfully. As mentioned, its a buffering issue. In fact, if the logger is busy/frequent, you won't notice the buffering issue ie you won't need the --line-buffered (or you can just be patient

).

Agrouf · 06-05-2008, 12:28 AM

Quote:

Originally Posted by billymayday

Just to clarify my posting - when I do

tail -f /var/log/maillog

I don't return to the command line prompt, hence my suggestion that it isn't actually anything to do with grep

This is normal behaviour I believe. tail -f doesn't return and shouldn't. If it returns, this is not normal.

billymayday · 06-05-2008, 12:51 AM

I think that's what I was trying to get to. This in turn would suggest that it's not a grep issue at all correct?

lostjohnny · 06-05-2008, 04:34 AM

Quote:

Originally Posted by arungoodboy

There's nothing a little googling can't solve! The root cause is grep's behaviour, not the pipes'. Apparently grep buffers the results so you won't (or may not) get the output instantaneously. The argument --line-buffered to the first grep solves your problem.

Code:

arun ~ $ tail -f temp.txt | grep --line-buffered 134 | grep SDSOffice
134 SDSOfficeTest blah
134 asdasdf SDSOfficeTest

Here's where I got the solution.

Thank you!

This makes sense to me now.

I'm still not entirely sure why grep (the first one) would buffer when its output is piped to another grep and not when the output is stdout, but I'm prepared to accept for now that it just does. Maybe it's something to do with how grep (the second one in this case) gets its input. It doesn't challenge my fundamental understanding of pipes, anyway.

Thanks again.

Lost Johhny

chrism01 · 06-05-2008, 06:42 PM

Both greps buffer (in fact all cmds do, seaprately), so the more cmds you pipe together, the bigger the effective buffer. Of course they flush when the cmd completes (although tail -f doesn't 'complete', as mentioned above).