need a script to continuously monitor appended data to a log file
 

Hello,

I need some help with the following script:

I have a continuously growing log file (x.log) in which i have to look for certain lines that contain "Long". The line above each line containing the word "Long" it contains a time stamp. I want to extract each line containing the time stamp into myfile.txt and check the difference between time stamps. Whenever there is a difference i need to run another script (${CAL_PATH}/${APP_NAME}), then sleep 1, then continue searching.

Lines with "Long" do not appear continuously, but in blasts. The script runs fine until the first pause encountered. Starting with the first pause, tail -f doesn't write in myfile.txt anymore.

Can someone help me understand why "tail -f" it stops writing into myfile.txt? Or does someone know an alternative to "tail -f" of achieving the initial scope of the script?

I'll appreciate any help.
Thanks!

My guess would be that the data is being buffered in the pipe and will not be seen by the reader awk or sed until tail has either emitted enough data to force a write to the pipe, or until the tail process completes. If you wait long enough, you may see some data emerge in the pipe. I don't see why you need to use tail at all.
Your use of the variable name $PATH seems questionable, as this is usually a standard variable used by the shell for its own well known purposes.

--- rod.

First of all thanks for your answer.

Second, sed is the actual reader of tail output. awk only reads the grep output, and i am using it to kill the previous started process "tail -f" when the log has been rotated, and i have to open and read the new log file.

About the $PATH it has nothing to do with the shell variable ... The actual script uses other variable names for defining the paths of the files and other file names of course. I just wanted to simplify names :).

Back to "tail -f", i need it because i have to read each line of the log file as this one is growing. If i am not able to read "realtime" from file, i might lose some of the lines which leads to not running the other script in the right moment, which screws everything up.:).

Can u help me with this?

Thanks

1. $PATH is a shell variable, do not use it for other purposes in your script.
2. "ps -fxu pin | grep ..." will (almost) always find one process: the grep command itself. So insert: "... | grep -v grep | ..."
3. The tail -f command is unlikely to stop working. It must be a buffering issue. Try to add "-u" to the sed command. But it will not help much, if the output pipeline of tail is already buffered (hopefully not).

Okay, I misread the pipe to awk part. I tried a mock-up of a similar scenario to yours: a process populates a file on a periodic basis, and 'tail -f' reads it into a pipe to sed. I'm not enough of a sed master to understand your script, but I get a feeling that the use of hold spaces might introduce some problem. Can you re-work the program to use a more pedestrian script that simply does a substitution or other simple edit, and see if that changes anything?
How is the input file built up? Is it opened-written-closed iteratively, or opened and held open continuously while being written? In my tests, I used a Perl script to update a file periodically. When I did the open()-write()-close() on every update, or when I used autoflush(1) on the filehandle, tail would see each update as it happened. If I put the open() & close() outside the write() loop, tail would not see the new data, and neither would the filesystem (using ls). Maybe the application that updates the log file writes chunks while holding the file open continuously. If you run tail -f on the file independently, do you get similar behavior?

I think that's my point. Since it is intended to be distinct from the well-known variable, it should have a name that is distinct, to avoid confusion.

--- rod.

Hey,


:) As i wrote before the actual variable used by the script is not $PATH (but $A_INTER_LOGS). A gave it a suggestive name when i posted the script in order to simplify the reading... Is may bad, i didn't think anyone would focus on that.

My real and only problem is with "tail -f".

I checked how the input file built up. There are many processes which write the log simultaneously, but each process opens the file, writes into it and then closes it. So, the file is not held open continuously while being written.

I tried "tail -f" outside the script and it works perfectly.
I can't understand what the problem is...:(

Thanks anyway for your effort!