LinuxQuestions.org
Did you know LQ has a Linux Hardware Compatibility List?
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 10-26-2010, 10:05 PM   #1
hattori.hanzo
Member
 
Registered: Aug 2006
Posts: 167

Rep: Reputation: 15
Process a log file on an hourly basis but the log only rotates once a day


I need to process a log file on an hourly basis but the log only rotates once a day.

Basically, I am trying to get the difference between the previous file and the current file based on datetime.

ie. The new file's datetime events > previous file's datetime events.

The first field in the files is datetime.

Code:
2010-10-27 01:57:32,aaa.bbb.ccc.ddd,host1
2010-10-27 01:57:32,aaa.bbb.ccc.ddd,host2
2010-10-27 01:57:38,aaa.bbb.ccc.ddd,host3
I tried using diff but since I need to take datetime into consideration this wont work.

Update: I can store the latest datetime stamp in a tmp file after the file is processed. Does awk or sed support a check which I can use this variable to compare another file and keep all entries > than the datetime stored?

Thanks & Regards,

Last edited by hattori.hanzo; 10-27-2010 at 01:46 AM.
 
Old 10-27-2010, 06:46 AM   #2
zhjim
Senior Member
 
Registered: Oct 2004
Distribution: Debian Squeeze x86_64
Posts: 1,436
Blog Entries: 11

Rep: Reputation: 181Reputation: 181
When you can get the line number you can tell awk as well as sed to start on this line. But I dunno how. But should not be hard to check how to do this.

Some other ideas on this.

1.) Hourly cron job (maybe calling a perl script)
1a) Get size of file (and write to file)
1b) Find the difference between the last file size and the new one
1c) Extract the lines
1d) Parse files

1.) Hourly cron job (bash script)
1a) grep $cur_date:$cur_hour-1 > /tmp/file
1b) Parse files

I once had a perl programm which did exactly what you need but can't find it anymore

P.S. Funny your LQ Post in fourth on google searching for "hourly parse log file linux"

Last edited by zhjim; 10-27-2010 at 06:47 AM.
 
1 members found this post helpful.
Old 10-27-2010, 03:59 PM   #3
hattori.hanzo
Member
 
Registered: Aug 2006
Posts: 167

Original Poster
Rep: Reputation: 15
Thanks for the response and suggestions.

I will try the line number idea first then the file size comparison.

I can't believe this post hasnt got more response as I would imagine it would be a common problem.
 
Old 10-27-2010, 04:27 PM   #4
zhjim
Senior Member
 
Registered: Oct 2004
Distribution: Debian Squeeze x86_64
Posts: 1,436
Blog Entries: 11

Rep: Reputation: 181Reputation: 181
Sad but true the common problems are to weary for most of the people...

Log parsing is one of the best habits of linux admins but also one of the hard-to-see-through stuff, cause there are just so many programs.
I'll dig on the prog I mentioned before to see how the name was exactly.
 
Old 10-28-2010, 08:55 AM   #5
hattori.hanzo
Member
 
Registered: Aug 2006
Posts: 167

Original Poster
Rep: Reputation: 15
What initially appears to be simple task is really quite difficult. I found this script which I have modified and appears to do what I need.

Hope this helps someone.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
In Apache server, How to change log file location and log format for access log fil? since1993 Linux - Server 1 08-19-2009 04:14 PM
/var/log/procmail.log.1 being mailed every day soylentgreen Linux - Server 3 07-10-2007 02:47 PM
Squid exits when rotates log eluzi Linux - Software 1 07-12-2006 04:30 AM
Get hourly log files - need help goboxe Programming 9 05-15-2006 09:20 PM
parse log file hourly onewave Programming 2 03-28-2005 01:52 PM


All times are GMT -5. The time now is 02:01 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration