need a script to continuously monitor appended data to a log file
ProgrammingThis forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
need a script to continuously monitor appended data to a log file
Hello,
I need some help with the following script:
Code:
if ! [ -f ${PATH}/myfile.txt ];then
echo $(date +%Y%m%d"_"%H%M%S)": Nu am gasit fisierul ${PATH}/myfile.txt"
ps -fxu pin | grep "/usr/local/coreutils/bin/tail -f ${LOG_PATH}/x.log$" | awk '{system("kill "$2)}'
cat ${LOG_PATH}/x.log | sed -n -e '/LONG/{x;1!p;g;;}' -e h > ${PATH}/myfile.txt
/usr/local/coreutils/bin/tail -f ${LOG_PATH}/x.log | sed -n -e '/LONG/{x;1!p;g;;}' -e h >> ${PATH}/myfile.txt &
fi
date1=$(tail -1 ${PATH}/myfile.txt | cut -d" " -f2-5)
if [ -f exec_long.date ];then
date2=$(cat exec_long.date)
else
date2=$date1
echo $date1 > exec_long.date
fi
if ! [ "$date1" = "$date2" ];then
${CAL_PATH}/${APP_NAME}
sleep 1
echo $date1 > exec_long.date
fi
I have a continuously growing log file (x.log) in which i have to look for certain lines that contain "Long". The line above each line containing the word "Long" it contains a time stamp. I want to extract each line containing the time stamp into myfile.txt and check the difference between time stamps. Whenever there is a difference i need to run another script (${CAL_PATH}/${APP_NAME}), then sleep 1, then continue searching.
Lines with "Long" do not appear continuously, but in blasts. The script runs fine until the first pause encountered. Starting with the first pause, tail -f doesn't write in myfile.txt anymore.
Can someone help me understand why "tail -f" it stops writing into myfile.txt? Or does someone know an alternative to "tail -f" of achieving the initial scope of the script?
My guess would be that the data is being buffered in the pipe and will not be seen by the reader awk or sed until tail has either emitted enough data to force a write to the pipe, or until the tail process completes. If you wait long enough, you may see some data emerge in the pipe. I don't see why you need to use tail at all.
Your use of the variable name $PATH seems questionable, as this is usually a standard variable used by the shell for its own well known purposes.
My guess would be that the data is being buffered in the pipe and will not be seen by the reader awk or sed until tail has either emitted enough data to force a write to the pipe, or until the tail process completes. If you wait long enough, you may see some data emerge in the pipe. I don't see why you need to use tail at all.
Your use of the variable name $PATH seems questionable, as this is usually a standard variable used by the shell for its own well known purposes.
--- rod.
First of all thanks for your answer.
Second, sed is the actual reader of tail output. awk only reads the grep output, and i am using it to kill the previous started process "tail -f" when the log has been rotated, and i have to open and read the new log file.
About the $PATH it has nothing to do with the shell variable ... The actual script uses other variable names for defining the paths of the files and other file names of course. I just wanted to simplify names .
Back to "tail -f", i need it because i have to read each line of the log file as this one is growing. If i am not able to read "realtime" from file, i might lose some of the lines which leads to not running the other script in the right moment, which screws everything up..
1. $PATH is a shell variable, do not use it for other purposes in your script.
2. "ps -fxu pin | grep ..." will (almost) always find one process: the grep command itself. So insert: "... | grep -v grep | ..."
3. The tail -f command is unlikely to stop working. It must be a buffering issue. Try to add "-u" to the sed command. But it will not help much, if the output pipeline of tail is already buffered (hopefully not).
Okay, I misread the pipe to awk part. I tried a mock-up of a similar scenario to yours: a process populates a file on a periodic basis, and 'tail -f' reads it into a pipe to sed. I'm not enough of a sed master to understand your script, but I get a feeling that the use of hold spaces might introduce some problem. Can you re-work the program to use a more pedestrian script that simply does a substitution or other simple edit, and see if that changes anything?
How is the input file built up? Is it opened-written-closed iteratively, or opened and held open continuously while being written? In my tests, I used a Perl script to update a file periodically. When I did the open()-write()-close() on every update, or when I used autoflush(1) on the filehandle, tail would see each update as it happened. If I put the open() & close() outside the write() loop, tail would not see the new data, and neither would the filesystem (using ls). Maybe the application that updates the log file writes chunks while holding the file open continuously. If you run tail -f on the file independently, do you get similar behavior?
Quote:
About the $PATH it has nothing to do with the shell variable
I think that's my point. Since it is intended to be distinct from the well-known variable, it should have a name that is distinct, to avoid confusion.
I think that's my point. Since it is intended to be distinct from the well-known variable, it should have a name that is distinct, to avoid confusion.
--- rod.
Hey,
As i wrote before the actual variable used by the script is not $PATH (but $A_INTER_LOGS). A gave it a suggestive name when i posted the script in order to simplify the reading... Is may bad, i didn't think anyone would focus on that.
My real and only problem is with "tail -f".
I checked how the input file built up. There are many processes which write the log simultaneously, but each process opens the file, writes into it and then closes it. So, the file is not held open continuously while being written.
I tried "tail -f" outside the script and it works perfectly.
I can't understand what the problem is...
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.