LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)
-   -   Scan log file for errors (https://www.linuxquestions.org/questions/linux-newbie-8/scan-log-file-for-errors-493341/)

huskerharry 10-17-2006 08:47 PM

Scan log file for errors
 
I am trying to write a script using BASH (Doesn't need to be in this) that will scan log files that are created by RSYNC jobs and look for "IO Error" wthin the logfiles. My RSYNC scripts create daily log files that are in this format: name`date +%Y%m%d`.log (Where name is the name of the script running).

I don't want the script to scan any files older than the current day though. I have seen suggestions of having a script use "tail" and just do a "tail -f" on a file, which would be good if my log files didn't change daily. Also, with this, is it possible that when this script is run, for it to remember that it had alreaday seen an error that may have previously happened in the log file that it is currently scanning?

Just quick info about my RSYNC scripts, they are synchronizing directories across a WAN (about 10GB- US to Australia and Australia to US), so it is kind of important that I can get notified that there is an issue that may have occured.

I have written something really basic that scans the entire directory of log files looking for "IO Error" in all the logs using "cat" and doing a "grep" on "IO Error", but as time goes and files get more abundant, this will not be feasable.

I would really appreciate any ideas, suggestions, and help with this.

haertig 10-17-2006 09:13 PM

As opposed to checking for errors in a logfile after-the-fact, why don't you check the exit value of rsync immediately after it completes? Anything non-zero would be suspicious. Run "man rsync" and scroll to near the end of the manpage (it's a long one!) to see a listing of exit values and what they mean.
Quote:

...scans the entire directory of log files looking for "IO Error" in all the logs using "cat" and doing a "grep" on "IO Error", but as time goes and files get more abundant, this will not be feasable.
This is still feasible, even with a lot of files. You just have to limit the file(s) you are looking at. Maybe by using "find -mtime ..."

huskerharry 10-17-2006 09:48 PM

haertig,

Thanks for that. I will give it a shot and let you know. :)

Won't "cat" on a fair few files cause processor grief though? I plan on only keeping maybe about a months worth of log files, then the rest will be in an archive folder, so those won't be searched. But still, the log files average in size from 9420 to 21424 (Not huge by any means), would "cat" effect performance?

chrism01 10-18-2006 02:51 AM

As he said the 2 classic approaches are:
1. Check rsync rtn codes : this is effectively realtime
2. use -mtime (or even -atime) options on find cmd. :this can be done post-facto (& repeated if needed).

haertig 10-18-2006 01:50 PM

Quote:

Originally Posted by huskerharry
But still, the log files average in size from 9420 to 21424 (Not huge by any means), would "cat" effect performance?

Performance degradation would be totally insignificant. Here's a 'cat' of a 5+ megabyte file (about 250 to 500 times the size of the files you gave as examples).
Code:

$ ls -l /boot/initrd.img-2.6.12-1-686
-rw-r--r--  1 root root 5308416 2005-10-24 08:57 /boot/initrd.img-2.6.12-1-686
$ time cat /boot/initrd.img-2.6.12-1-686 > /dev/null

real    0m0.018s
user    0m0.001s
sys    0m0.004s
$

I think 0.004 seconds of system time would be largely unnoticeable to all but the most critical computer user!

huskerharry 10-19-2006 01:35 AM

Thank you both for the information. For the time being i have written something easy (A pain in the butt)that will scan the logs but doesn't output what file it finds stuff in. I unfortuantely have had to concentrate on a Company name change yesterday and today, but want to try the suggestions that you have mentioned. I really appreciate all the help and assistance!!!

Here is my "Quick/Simple/temp" fix:

Code:

cat /usr/company/log/rsync/*.log | grep "code=" | grep  -v "code=0" > \
/usr/company/log/rsync/errors`date +%Y%m%d`.txt


Emmanuel_uk 10-19-2006 02:07 AM

would swatch do what you want?

http://linux.maruhn.com/sec/swatch.html

also logwatcher?

huskerharry 10-20-2006 01:43 AM

Thanks, I will have a look into those as I am not familiar with them.

I would prefer not to install programs, but they are definitely worth having a look at though. :)


All times are GMT -5. The time now is 12:32 PM.