Linux - NewbieThis Linux forum is for members that are new to Linux.
Just starting out and have a question?
If it is not in the man pages or the how-to's this is the place!
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
I am trying to write a script using BASH (Doesn't need to be in this) that will scan log files that are created by RSYNC jobs and look for "IO Error" wthin the logfiles. My RSYNC scripts create daily log files that are in this format: name`date +%Y%m%d`.log (Where name is the name of the script running).
I don't want the script to scan any files older than the current day though. I have seen suggestions of having a script use "tail" and just do a "tail -f" on a file, which would be good if my log files didn't change daily. Also, with this, is it possible that when this script is run, for it to remember that it had alreaday seen an error that may have previously happened in the log file that it is currently scanning?
Just quick info about my RSYNC scripts, they are synchronizing directories across a WAN (about 10GB- US to Australia and Australia to US), so it is kind of important that I can get notified that there is an issue that may have occured.
I have written something really basic that scans the entire directory of log files looking for "IO Error" in all the logs using "cat" and doing a "grep" on "IO Error", but as time goes and files get more abundant, this will not be feasable.
I would really appreciate any ideas, suggestions, and help with this.
As opposed to checking for errors in a logfile after-the-fact, why don't you check the exit value of rsync immediately after it completes? Anything non-zero would be suspicious. Run "man rsync" and scroll to near the end of the manpage (it's a long one!) to see a listing of exit values and what they mean.
...scans the entire directory of log files looking for "IO Error" in all the logs using "cat" and doing a "grep" on "IO Error", but as time goes and files get more abundant, this will not be feasable.
This is still feasible, even with a lot of files. You just have to limit the file(s) you are looking at. Maybe by using "find -mtime ..."
Thanks for that. I will give it a shot and let you know.
Won't "cat" on a fair few files cause processor grief though? I plan on only keeping maybe about a months worth of log files, then the rest will be in an archive folder, so those won't be searched. But still, the log files average in size from 9420 to 21424 (Not huge by any means), would "cat" effect performance?
As he said the 2 classic approaches are:
1. Check rsync rtn codes : this is effectively realtime
2. use -mtime (or even -atime) options on find cmd. :this can be done post-facto (& repeated if needed).
Thank you both for the information. For the time being i have written something easy (A pain in the butt)that will scan the logs but doesn't output what file it finds stuff in. I unfortuantely have had to concentrate on a Company name change yesterday and today, but want to try the suggestions that you have mentioned. I really appreciate all the help and assistance!!!