how to monitor logfiles

ganninu · 12-05-2003, 08:21 AM

This thread is directly related to thread http://www.linuxquestions.org/questi...hreadid=122565

Are there any techniques which can be usedto process growing logfiles? I have this logfile and I'm trying to monitor for new entries to process them.... For some strange reason,

tail -1 | <process_entry.bash>

works perfectly well for the last entry, but:

tail -f | <process_entry.bash>

doesn't give any output at all!!!

Any techniques how to go around this?? Help very much appreciated.

Ganninu.

druuna · 12-05-2003, 11:53 AM

I can explain to you why it behaves this way:

<command 1> | <command 2>

- The shell creates 2 subprocesses (fork), lets call them sp1 and sp2.
- Set up a pipe (I/O) between sp1's output and sp2's input.
- Start (exec) <command 1> in process sp1.
- Start (exec) <command 2> in process sp2.
- Wait for both prcesses to finish.

The last line is the reason why this will work with tail -1 and _not_ with tail -f. The shell will wait untill both processes are ready, tail -f will never be ready, hench: no output.

If you put a trace on your script you will get confirmation for this behavior:

rt_sigaction(SIGINT, {0x80701a0, [], 0x4000000}, {SIG_DFL}, 8) = 0
wait4(-1,

No more output after the above 2 lines. Shell is waiting for one of the processes (the tail -f in this case) to finish.

I don't know how to avoid this (restarting syslogd everytime you run the script won't be an option

, but will work).

You could consider to fetch all the lines from the logfile that haven't been processesed (based on timestamps), put these in a seperate file and use that file.

druuna · 12-06-2003, 09:53 AM

Had some more time to look at your problem and maybe this will work for you.

Code:

#!/bin/bash
tail -f /var/log/messages | \
while read line
do
  echo $line | grep 'session' | awk '{ print $7, $8, $9, $11 }' >> /var/log/out.file
done

The above code will look for lines containing 'session' and print the following to the out.file if it finds a hit:

session started for root,

The above example is quick and dirty, but shows that this construction will work.

ganninu · 12-06-2003, 10:09 AM

thanks i'll give it a look and apply it to my needs and tell you what happened

markus1982 · 12-07-2003, 07:36 AM

take a look at "logcheck" also :-)