[SOLVED] Process a log file on an hourly basis but the log only rotates once a day

hattori.hanzo · 10-26-2010, 10:05 PM

I need to process a log file on an hourly basis but the log only rotates once a day.

Basically, I am trying to get the difference between the previous file and the current file based on datetime.

ie. The new file's datetime events > previous file's datetime events.

The first field in the files is datetime.

Code:

2010-10-27 01:57:32,aaa.bbb.ccc.ddd,host1
2010-10-27 01:57:32,aaa.bbb.ccc.ddd,host2
2010-10-27 01:57:38,aaa.bbb.ccc.ddd,host3

I tried using diff but since I need to take datetime into consideration this wont work.

Update: I can store the latest datetime stamp in a tmp file after the file is processed. Does awk or sed support a check which I can use this variable to compare another file and keep all entries > than the datetime stored?

Thanks & Regards,

zhjim · 10-27-2010, 06:46 AM

When you can get the line number you can tell awk as well as sed to start on this line. But I dunno how. But should not be hard to check how to do this.

Some other ideas on this.

1.) Hourly cron job (maybe calling a perl script)
1a) Get size of file (and write to file)
1b) Find the difference between the last file size and the new one
1c) Extract the lines
1d) Parse files

1.) Hourly cron job (bash script)
1a) grep $cur_date:$cur_hour-1 > /tmp/file
1b) Parse files

I once had a perl programm which did exactly what you need but can't find it anymore

P.S. Funny your LQ Post in fourth on google searching for "hourly parse log file linux"

hattori.hanzo · 10-27-2010, 03:59 PM

Thanks for the response and suggestions.

I will try the line number idea first then the file size comparison.

I can't believe this post hasnt got more response as I would imagine it would be a common problem.

zhjim · 10-27-2010, 04:27 PM

Sad but true the common problems are to weary for most of the people...

Log parsing is one of the best habits of linux admins but also one of the hard-to-see-through stuff, cause there are just so many programs.
I'll dig on the prog I mentioned before to see how the name was exactly.

hattori.hanzo · 10-28-2010, 08:55 AM

What initially appears to be simple task is really quite difficult. I found this script which I have modified and appears to do what I need.

Hope this helps someone.