LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 10-13-2010, 06:51 AM   #1
akitty
LQ Newbie
 
Registered: Nov 2008
Posts: 9

Rep: Reputation: 0
Unhappy need a script to continuously monitor appended data to a log file


Hello,

I need some help with the following script:

Code:
if ! [ -f ${PATH}/myfile.txt ];then
  echo $(date +%Y%m%d"_"%H%M%S)": Nu am gasit fisierul ${PATH}/myfile.txt"
  ps -fxu pin | grep "/usr/local/coreutils/bin/tail -f ${LOG_PATH}/x.log$" | awk '{system("kill "$2)}'
  cat ${LOG_PATH}/x.log | sed -n -e '/LONG/{x;1!p;g;;}' -e h > ${PATH}/myfile.txt
  
  /usr/local/coreutils/bin/tail -f ${LOG_PATH}/x.log | sed -n -e '/LONG/{x;1!p;g;;}' -e h >> ${PATH}/myfile.txt &
fi
date1=$(tail -1 ${PATH}/myfile.txt | cut -d" " -f2-5)
if [ -f exec_long.date ];then
  date2=$(cat exec_long.date)
else
  date2=$date1
  echo $date1 > exec_long.date
fi
if ! [ "$date1" = "$date2" ];then
        ${CAL_PATH}/${APP_NAME}
        sleep 1
        echo $date1 > exec_long.date
fi
I have a continuously growing log file (x.log) in which i have to look for certain lines that contain "Long". The line above each line containing the word "Long" it contains a time stamp. I want to extract each line containing the time stamp into myfile.txt and check the difference between time stamps. Whenever there is a difference i need to run another script (${CAL_PATH}/${APP_NAME}), then sleep 1, then continue searching.

Lines with "Long" do not appear continuously, but in blasts. The script runs fine until the first pause encountered. Starting with the first pause, tail -f doesn't write in myfile.txt anymore.

Can someone help me understand why "tail -f" it stops writing into myfile.txt? Or does someone know an alternative to "tail -f" of achieving the initial scope of the script?

I'll appreciate any help.
Thanks!
 
Old 10-13-2010, 09:09 AM   #2
theNbomr
LQ 5k Club
 
Registered: Aug 2005
Distribution: OpenSuse, Fedora, Redhat, Debian
Posts: 5,398
Blog Entries: 2

Rep: Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908
My guess would be that the data is being buffered in the pipe and will not be seen by the reader awk or sed until tail has either emitted enough data to force a write to the pipe, or until the tail process completes. If you wait long enough, you may see some data emerge in the pipe. I don't see why you need to use tail at all.
Your use of the variable name $PATH seems questionable, as this is usually a standard variable used by the shell for its own well known purposes.

--- rod.

Last edited by theNbomr; 10-13-2010 at 09:12 AM.
 
Old 10-13-2010, 09:35 AM   #3
akitty
LQ Newbie
 
Registered: Nov 2008
Posts: 9

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by theNbomr View Post
My guess would be that the data is being buffered in the pipe and will not be seen by the reader awk or sed until tail has either emitted enough data to force a write to the pipe, or until the tail process completes. If you wait long enough, you may see some data emerge in the pipe. I don't see why you need to use tail at all.
Your use of the variable name $PATH seems questionable, as this is usually a standard variable used by the shell for its own well known purposes.

--- rod.
First of all thanks for your answer.

Second, sed is the actual reader of tail output. awk only reads the grep output, and i am using it to kill the previous started process "tail -f" when the log has been rotated, and i have to open and read the new log file.

About the $PATH it has nothing to do with the shell variable ... The actual script uses other variable names for defining the paths of the files and other file names of course. I just wanted to simplify names .

Back to "tail -f", i need it because i have to read each line of the log file as this one is growing. If i am not able to read "realtime" from file, i might lose some of the lines which leads to not running the other script in the right moment, which screws everything up..

Can u help me with this?

Thanks
 
Old 10-13-2010, 04:21 PM   #4
J_Szucs
Senior Member
 
Registered: Nov 2001
Location: Budapest, Hungary
Distribution: SuSE 6.4-11.3, Dsl linux, FreeBSD 4.3-6.2, Mandrake 8.2, Redhat, UHU, Debian Etch
Posts: 1,126

Rep: Reputation: 58
1. $PATH is a shell variable, do not use it for other purposes in your script.
2. "ps -fxu pin | grep ..." will (almost) always find one process: the grep command itself. So insert: "... | grep -v grep | ..."
3. The tail -f command is unlikely to stop working. It must be a buffering issue. Try to add "-u" to the sed command. But it will not help much, if the output pipeline of tail is already buffered (hopefully not).
 
Old 10-13-2010, 06:00 PM   #5
theNbomr
LQ 5k Club
 
Registered: Aug 2005
Distribution: OpenSuse, Fedora, Redhat, Debian
Posts: 5,398
Blog Entries: 2

Rep: Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908
Okay, I misread the pipe to awk part. I tried a mock-up of a similar scenario to yours: a process populates a file on a periodic basis, and 'tail -f' reads it into a pipe to sed. I'm not enough of a sed master to understand your script, but I get a feeling that the use of hold spaces might introduce some problem. Can you re-work the program to use a more pedestrian script that simply does a substitution or other simple edit, and see if that changes anything?
How is the input file built up? Is it opened-written-closed iteratively, or opened and held open continuously while being written? In my tests, I used a Perl script to update a file periodically. When I did the open()-write()-close() on every update, or when I used autoflush(1) on the filehandle, tail would see each update as it happened. If I put the open() & close() outside the write() loop, tail would not see the new data, and neither would the filesystem (using ls). Maybe the application that updates the log file writes chunks while holding the file open continuously. If you run tail -f on the file independently, do you get similar behavior?

Quote:
About the $PATH it has nothing to do with the shell variable
I think that's my point. Since it is intended to be distinct from the well-known variable, it should have a name that is distinct, to avoid confusion.

--- rod.
 
Old 10-14-2010, 02:39 AM   #6
akitty
LQ Newbie
 
Registered: Nov 2008
Posts: 9

Original Poster
Rep: Reputation: 0
Smile

Quote:
Originally Posted by theNbomr View Post

I think that's my point. Since it is intended to be distinct from the well-known variable, it should have a name that is distinct, to avoid confusion.

--- rod.

Hey,


As i wrote before the actual variable used by the script is not $PATH (but $A_INTER_LOGS). A gave it a suggestive name when i posted the script in order to simplify the reading... Is may bad, i didn't think anyone would focus on that.

My real and only problem is with "tail -f".

I checked how the input file built up. There are many processes which write the log simultaneously, but each process opens the file, writes into it and then closes it. So, the file is not held open continuously while being written.

I tried "tail -f" outside the script and it works perfectly.
I can't understand what the problem is...

Thanks anyway for your effort!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Daemon shell or perl script to monitor a log file khriz Programming 4 01-07-2010 07:35 AM
Need Help For this Monitor Log Script dellxmax Solaris / OpenSolaris 4 11-14-2009 01:09 PM
Shell script to Continuously scan a log file for success novice82 Linux - Newbie 4 10-09-2009 02:20 AM
restore a tape that has appended data on it blckspder Linux - Server 1 09-29-2008 04:11 PM
shell script to monitor log file calipryss Linux - Newbie 14 08-05-2008 10:46 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 11:18 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration