Selecting last 60 minutes from log files

WojtekO · 04-09-2009, 03:32 PM

Hello there,

I'm monitoring 20 asterisk servers and I've developed a small php utility that displays me their errors when they happen.

Currently I'm using the something like this:
tail -5000 /var/log/asterisk/full | egrep -i "ERROR|WARNING" > errors.log

The problem with this is that some servers are getting more traffic then others resulting that on one box 5000 lines might last 5 hours and on another 5 minutes. And as such what I see on the screen might be outdated by a few hours or scrolls too fast for me to see.

I'd like to know what would be the most efficient command to extract only the last 60 minutes and perform my egrep on that. Cron is setup every minute.

Here's a sample of the log format:

Code:

[Apr  9 08:21:38] VERBOSE[13499] logger.c:  Found request for channel [1239279644.1537]
[Apr  9 08:21:38] VERBOSE[13499] logger.c:  CTI Asked for EXECAPP on channel [Zap/16-1], App [SBR_PLAY_ANNOUNCEMENT], Params [B310_CSMR_DAY_Fr_Dec08], confirmRequest [0]
[Apr  9 08:21:38] VERBOSE[13499] logger.c:  Searching CallRoute with tracknum [1239279666.1543]
[Apr  9 08:21:38] VERBOSE[13499] logger.c:  Found request for channel [1239279666.1543]
[Apr  9 08:21:39] VERBOSE[13499] logger.c:  CTI Asked for EXECAPP on channel [Zap/16-1], App [MusicOnHold], Params [], confirmRequest [0]
[Apr  9 08:21:39] VERBOSE[13499] logger.c:  Searching CallRoute with tracknum [1239279666.1543]

I've googled a bit but the things I found do not work.
For example:
grep -A 5 "Apr 9 08:21:38" full
Should return 5 lines after the match, but all it does it matches every "Apr 9 08:21:38" like a normal grep.

I wanted to do some kind of `date --date="1 hours ago"` and format that to feed the script so it selects everything below that var. And egrep the rest for errors.

Any input appreciated. Thank You

i92guboj · 04-09-2009, 04:04 PM

Unfortunately, unless you can find an efective way to match by contents there's no way around this. What you want to do would only be possible if every single byte in the file had an mtime. mtime can only tell you when a given file was modified, but how to determine which contents was added is another thing, a completely different one. You can grep for patterns inside the file (well, grep or awk or whatever), but nothing else. Some tools might need extra configuration or even patching to print a date at the beginning of each line when logging so you can grep for them.

WojtekO · 04-09-2009, 11:17 PM

I could build a dirty script like this:

script.sh

Code:

1min = `date --date="1 minutes ago"`
2min = `date --date="2 minutes ago"`
[...] (3 to 58)
59min = `date --date="59 minutes ago"`
60min = `date --date="60 minutes ago"`

tail -100000 /var/log/asterisk/full | egrep -i "$1min|$2min|[...]|$59min|$60min" | egrep -i "ERROR|WARNING" > errors.log

But I was wondering if there was a more elegant and efficient way to do the above.

Quote:

Originally Posted by i92guboj

Some tools might need extra configuration or even patching to print a date at the beginning of each line when logging so you can grep for them.

The logs do already contain the date stamp in them. How would you suggest I 'grep for them'. Like in my sample code above?

chrism01 · 04-10-2009, 12:04 AM

Basically you need a lang that handles date calcs easily; I recommend Perl. Loads of pre-built modules/cmds for that.
http://perldoc.perl.org/. Its a very popular lang for sysadmins.

ghostdog74 · 04-10-2009, 02:25 AM

Code:

tail -5000 /var/log/asterisk/full | awk -F"] " 'BEGIN{
 min=3600 #3600 in 60mins
 m["Jan"]="01"
 m["Feb"]="02"
 m["Mar"]="03"
 m["Apr"]="04"
 m["May"]="05"
 m["Jun"]="06"
 m["Jul"]="07"
 m["Aug"]="08"
 m["Sep"]="09"
 m["Oct"]="10"
 m["Nov"]="11"
 m["Dec"]="12"
 today=systime() 
 year=2009
}
{ 
 sub(/\[/,"",$1)
 o=split($1,DATETIME," ")
 mth=m[DATETIME[1]]
 day=DATETIME[2]
 time=DATETIME[3]
 n=split(time,t,":")
 hour=t[1]
 mm = t[2]
 sec= t[3]
 d = year" "mth" "day" "hour" "mm" "sec
 if (  (today - mktime(d) ) <= min ) {
   if ( $0 ~ /ERROR|WARNING/){
     print $0 
  }
 }
}'

syg00 · 04-10-2009, 02:29 AM

What about passing midnight/end-of-year/daylight saving ... ? (yes, I know that isn't a problem here in Brisbane).
I'd second the perl suggestion.

I tried some bash options, but the midnight thing got ugly.