LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 04-09-2009, 03:32 PM   #1
WojtekO
Member
 
Registered: May 2006
Distribution: CentOS 5
Posts: 47

Rep: Reputation: 15
Selecting last 60 minutes from log files


Hello there,

I'm monitoring 20 asterisk servers and I've developed a small php utility that displays me their errors when they happen.

Currently I'm using the something like this:
tail -5000 /var/log/asterisk/full | egrep -i "ERROR|WARNING" > errors.log

The problem with this is that some servers are getting more traffic then others resulting that on one box 5000 lines might last 5 hours and on another 5 minutes. And as such what I see on the screen might be outdated by a few hours or scrolls too fast for me to see.

I'd like to know what would be the most efficient command to extract only the last 60 minutes and perform my egrep on that. Cron is setup every minute.

Here's a sample of the log format:
Code:
[Apr  9 08:21:38] VERBOSE[13499] logger.c:  Found request for channel [1239279644.1537]
[Apr  9 08:21:38] VERBOSE[13499] logger.c:  CTI Asked for EXECAPP on channel [Zap/16-1], App [SBR_PLAY_ANNOUNCEMENT], Params [B310_CSMR_DAY_Fr_Dec08], confirmRequest [0]
[Apr  9 08:21:38] VERBOSE[13499] logger.c:  Searching CallRoute with tracknum [1239279666.1543]
[Apr  9 08:21:38] VERBOSE[13499] logger.c:  Found request for channel [1239279666.1543]
[Apr  9 08:21:39] VERBOSE[13499] logger.c:  CTI Asked for EXECAPP on channel [Zap/16-1], App [MusicOnHold], Params [], confirmRequest [0]
[Apr  9 08:21:39] VERBOSE[13499] logger.c:  Searching CallRoute with tracknum [1239279666.1543]
I've googled a bit but the things I found do not work.
For example:
grep -A 5 "Apr 9 08:21:38" full
Should return 5 lines after the match, but all it does it matches every "Apr 9 08:21:38" like a normal grep.

I wanted to do some kind of `date --date="1 hours ago"` and format that to feed the script so it selects everything below that var. And egrep the rest for errors.

Any input appreciated. Thank You

Last edited by WojtekO; 04-09-2009 at 03:36 PM.
 
Old 04-09-2009, 04:04 PM   #2
i92guboj
Gentoo support team
 
Registered: May 2008
Location: Lucena, Córdoba (Spain)
Distribution: Gentoo
Posts: 4,083

Rep: Reputation: 405Reputation: 405Reputation: 405Reputation: 405Reputation: 405
Unfortunately, unless you can find an efective way to match by contents there's no way around this. What you want to do would only be possible if every single byte in the file had an mtime. mtime can only tell you when a given file was modified, but how to determine which contents was added is another thing, a completely different one. You can grep for patterns inside the file (well, grep or awk or whatever), but nothing else. Some tools might need extra configuration or even patching to print a date at the beginning of each line when logging so you can grep for them.
 
Old 04-09-2009, 11:17 PM   #3
WojtekO
Member
 
Registered: May 2006
Distribution: CentOS 5
Posts: 47

Original Poster
Rep: Reputation: 15
I could build a dirty script like this:

script.sh

Code:
1min = `date --date="1 minutes ago"`
2min = `date --date="2 minutes ago"`
[...] (3 to 58)
59min = `date --date="59 minutes ago"`
60min = `date --date="60 minutes ago"`

tail -100000 /var/log/asterisk/full | egrep -i "$1min|$2min|[...]|$59min|$60min" | egrep -i "ERROR|WARNING" > errors.log
But I was wondering if there was a more elegant and efficient way to do the above.

Quote:
Originally Posted by i92guboj View Post
Some tools might need extra configuration or even patching to print a date at the beginning of each line when logging so you can grep for them.
The logs do already contain the date stamp in them. How would you suggest I 'grep for them'. Like in my sample code above?
 
Old 04-10-2009, 12:04 AM   #4
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Rocky 9.2
Posts: 18,348

Rep: Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749
Basically you need a lang that handles date calcs easily; I recommend Perl. Loads of pre-built modules/cmds for that.
http://perldoc.perl.org/. Its a very popular lang for sysadmins.
 
Old 04-10-2009, 02:25 AM   #5
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,697
Blog Entries: 5

Rep: Reputation: 244Reputation: 244Reputation: 244
Code:
tail -5000 /var/log/asterisk/full | awk -F"] " 'BEGIN{
 min=3600 #3600 in 60mins
 m["Jan"]="01"
 m["Feb"]="02"
 m["Mar"]="03"
 m["Apr"]="04"
 m["May"]="05"
 m["Jun"]="06"
 m["Jul"]="07"
 m["Aug"]="08"
 m["Sep"]="09"
 m["Oct"]="10"
 m["Nov"]="11"
 m["Dec"]="12"
 today=systime() 
 year=2009
}
{ 
 sub(/\[/,"",$1)
 o=split($1,DATETIME," ")
 mth=m[DATETIME[1]]
 day=DATETIME[2]
 time=DATETIME[3]
 n=split(time,t,":")
 hour=t[1]
 mm = t[2]
 sec= t[3]
 d = year" "mth" "day" "hour" "mm" "sec
 if (  (today - mktime(d) ) <= min ) {
   if ( $0 ~ /ERROR|WARNING/){
     print $0 
  }
 }
}'
 
Old 04-10-2009, 02:29 AM   #6
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,103

Rep: Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117
What about passing midnight/end-of-year/daylight saving ... ? (yes, I know that isn't a problem here in Brisbane).
I'd second the perl suggestion.

I tried some bash options, but the midnight thing got ugly.

Last edited by syg00; 04-10-2009 at 02:30 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
What the %$#@ is pam_unix (cron:session) doing every ten minutes? (/var/log/auth.log) CoffeeKing!!! Linux - Security 3 02-05-2009 07:07 AM
Router log shows linux machine getting a new lease same ip every 20 minutes cknowles Linux - Newbie 3 12-24-2008 03:12 PM
selecting all files to mv chisholm Linux - Newbie 2 11-09-2008 03:40 PM
bg darkens on selecting log out allelopath Linux - Software 0 10-06-2005 08:22 PM
Can log files be time stamped? (such as FTP login and transfer log files) bripage Linux - Networking 6 08-08-2002 10:55 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 01:19 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration