LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices



Reply
 
Search this Thread
Old 10-17-2006, 09:47 PM   #1
huskerharry
LQ Newbie
 
Registered: Oct 2006
Posts: 5

Rep: Reputation: 0
Question Scan log file for errors


I am trying to write a script using BASH (Doesn't need to be in this) that will scan log files that are created by RSYNC jobs and look for "IO Error" wthin the logfiles. My RSYNC scripts create daily log files that are in this format: name`date +%Y%m%d`.log (Where name is the name of the script running).

I don't want the script to scan any files older than the current day though. I have seen suggestions of having a script use "tail" and just do a "tail -f" on a file, which would be good if my log files didn't change daily. Also, with this, is it possible that when this script is run, for it to remember that it had alreaday seen an error that may have previously happened in the log file that it is currently scanning?

Just quick info about my RSYNC scripts, they are synchronizing directories across a WAN (about 10GB- US to Australia and Australia to US), so it is kind of important that I can get notified that there is an issue that may have occured.

I have written something really basic that scans the entire directory of log files looking for "IO Error" in all the logs using "cat" and doing a "grep" on "IO Error", but as time goes and files get more abundant, this will not be feasable.

I would really appreciate any ideas, suggestions, and help with this.
 
Old 10-17-2006, 10:13 PM   #2
haertig
Senior Member
 
Registered: Nov 2004
Distribution: Debian, Ubuntu, LinuxMint, Slackware, SysrescueCD
Posts: 2,032

Rep: Reputation: 309Reputation: 309Reputation: 309Reputation: 309
As opposed to checking for errors in a logfile after-the-fact, why don't you check the exit value of rsync immediately after it completes? Anything non-zero would be suspicious. Run "man rsync" and scroll to near the end of the manpage (it's a long one!) to see a listing of exit values and what they mean.
Quote:
...scans the entire directory of log files looking for "IO Error" in all the logs using "cat" and doing a "grep" on "IO Error", but as time goes and files get more abundant, this will not be feasable.
This is still feasible, even with a lot of files. You just have to limit the file(s) you are looking at. Maybe by using "find -mtime ..."
 
Old 10-17-2006, 10:48 PM   #3
huskerharry
LQ Newbie
 
Registered: Oct 2006
Posts: 5

Original Poster
Rep: Reputation: 0
haertig,

Thanks for that. I will give it a shot and let you know.

Won't "cat" on a fair few files cause processor grief though? I plan on only keeping maybe about a months worth of log files, then the rest will be in an archive folder, so those won't be searched. But still, the log files average in size from 9420 to 21424 (Not huge by any means), would "cat" effect performance?
 
Old 10-18-2006, 03:51 AM   #4
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.6, Centos 5.10
Posts: 16,324

Rep: Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041
As he said the 2 classic approaches are:
1. Check rsync rtn codes : this is effectively realtime
2. use -mtime (or even -atime) options on find cmd. :this can be done post-facto (& repeated if needed).
 
Old 10-18-2006, 02:50 PM   #5
haertig
Senior Member
 
Registered: Nov 2004
Distribution: Debian, Ubuntu, LinuxMint, Slackware, SysrescueCD
Posts: 2,032

Rep: Reputation: 309Reputation: 309Reputation: 309Reputation: 309
Quote:
Originally Posted by huskerharry
But still, the log files average in size from 9420 to 21424 (Not huge by any means), would "cat" effect performance?
Performance degradation would be totally insignificant. Here's a 'cat' of a 5+ megabyte file (about 250 to 500 times the size of the files you gave as examples).
Code:
$ ls -l /boot/initrd.img-2.6.12-1-686
-rw-r--r--  1 root root 5308416 2005-10-24 08:57 /boot/initrd.img-2.6.12-1-686
$ time cat /boot/initrd.img-2.6.12-1-686 > /dev/null

real    0m0.018s
user    0m0.001s
sys     0m0.004s
$
I think 0.004 seconds of system time would be largely unnoticeable to all but the most critical computer user!

Last edited by haertig; 10-18-2006 at 02:51 PM.
 
Old 10-19-2006, 02:35 AM   #6
huskerharry
LQ Newbie
 
Registered: Oct 2006
Posts: 5

Original Poster
Rep: Reputation: 0
Thank you both for the information. For the time being i have written something easy (A pain in the butt)that will scan the logs but doesn't output what file it finds stuff in. I unfortuantely have had to concentrate on a Company name change yesterday and today, but want to try the suggestions that you have mentioned. I really appreciate all the help and assistance!!!

Here is my "Quick/Simple/temp" fix:

Code:
cat /usr/company/log/rsync/*.log | grep "code=" | grep  -v "code=0" > \
/usr/company/log/rsync/errors`date +%Y%m%d`.txt
 
Old 10-19-2006, 03:07 AM   #7
Emmanuel_uk
Senior Member
 
Registered: Nov 2004
Distribution: Mandriva mostly, vector 5.1, tried many.Suse gone from HD because bad Novell/Zinblows agreement
Posts: 1,604

Rep: Reputation: 47
would swatch do what you want?

http://linux.maruhn.com/sec/swatch.html

also logwatcher?
 
Old 10-20-2006, 02:43 AM   #8
huskerharry
LQ Newbie
 
Registered: Oct 2006
Posts: 5

Original Poster
Rep: Reputation: 0
Thanks, I will have a look into those as I am not familiar with them.

I would prefer not to install programs, but they are definitely worth having a look at though.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
scan for sector errors on hard disks psirek Linux - Software 1 04-19-2006 05:58 PM
write errors to log file, instead of /dev/null c_r2005 Linux - General 2 07-30-2005 09:34 AM
How do I log on-screen errors to file skunkburner Linux - General 9 06-02-2005 03:51 AM
error: stat of /var/log/ppp/connect-errors failed: No such file or directory Niceman2005 Linux - Newbie 0 01-20-2005 01:29 AM
F-prot Anti-virus scan log, suspicous file question. webwolf70 Linux - Security 5 11-16-2004 10:15 AM


All times are GMT -5. The time now is 01:09 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration