[SOLVED] counting 503 errors in a log for the last hour...

Habitual · 11-27-2013, 01:21 PM

I need to acquire the total number of 503 errors in a log file for the last hour.

I used this to get some particulars from the log file:
grep "HTTP/1.1\" 503" /mnt/varnish/varnish-access.log | tail -1

Code:

174.76.140.74 - - [27/Nov/2013:08:48:51 -0800] "GET http://secondary-schools.domain.in/rx/admin.emp.735.js HTTP/1.1" 503 435 "http://secondary-schools.domain.in/" "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.57 Safari/537.36

yes, that IS a domain.in (sanitized of course).

What I have worked out are the dates, as I planned to use bash to do this, but I think I'm using the wrong tool for the job.

Code:

date '+%d/%b/%Y:%I:%M:%S'
date '+%d/%b/%Y:%I:%M:%S' --date="1 hour ago"

"'now' minus 1 hour" is the requirement and the ‘hour’ and the ‘count' part should be "configurable" (allow the client, or us to readily adjust these 2 options)

I can't use grep with a from/to range (can I?) and sed is only good for replace in a from/to format, so I concluded that awk (or perl) must be the choice and I know neither.

I'd appreciate any direction, or pointers to a working solution.

Thank you for your time.

unSpawn · 11-27-2013, 01:50 PM

How about

Code:

logwatch --output stdout --numeric --service http --detail Medium --range 'since 1 hours ago for those hours'|grep 503 -A1

Habitual · 11-27-2013, 01:54 PM

Not installed, and I'm not sure I can.
But thanks!

danielbmartin · 11-27-2013, 02:24 PM

Quote:

Originally Posted by Habitual

I need to acquire the total number of 503 errors in a log file for the last hour.

This task is simplified if your one-hour "window" may be defined as falling on neat hourly bounds. For example, 14:00:00 to 14:59:59 but not 14:17:05 to 15:17:04.

Daniel B. Martin

Habitual · 11-27-2013, 02:28 PM

Daniel:

I asked that very thing too!
"in the last hour" I would grep and count from 0700 - 0800
otherwise
grep and count as "'now' - 1 hour" and he replied the latter.

Thanks.

Habitual · 11-29-2013, 10:43 AM

Quote:

Originally Posted by unSpawn

How about

Code:

logwatch --output stdout --numeric --service http --detail Medium --range 'since 1 hours ago for those hours'|grep 503 -A1

Thanks UnSpawn, I installed this from source (oneric whined about broken deps) on 1 of the clients' hosts.
I am reading up on it now.

Habitual · 12-02-2013, 08:59 AM

https://www.linuxquestions.org/quest...er-4175486549/

Habitual · 12-02-2013, 11:15 AM

Code:

#!/bin/bash
THISHOUR=$(date '+%I:%M')
LASTHOUR=$(date '+%I:%M' --date="1 hour ago")
sed -n "/$LASTHOUR/,/$THISHOUR/p" /mnt/varnish/varnish-access.log | grep "HTTP/1.1\" 503"
#EOF

real 0m11.342s
vs
logwatch --service varnish --range today --detail medium --print and > 15m = no brainer!

Thanks unSpawn.
You were just the motivation I needed.

unSpawn · 12-02-2013, 01:37 PM

Thanks for the feedback!

Habitual · 12-02-2013, 01:44 PM

easy-peasy,
mac-n-cheesy!

Habitual · 12-04-2013, 10:08 AM

Well, it was a great exercise in bash date|time skills.
But I knew there was a varnish-related tool to do the job and there is:

Code:

varnishtop -1 -i TxStatus | grep 503

Well, the client payed for the 'experience'.

Habitual · 12-05-2013, 03:22 PM

Re-worked it to include the UTC timestamp. I hate TZ conversions!

Code:

#!/bin/bash
# JJ of cirrhus9.com
# Wed Dec  4 16:21:39 EST
# cron: * * * * * /etc/zabbix/count_varnish_503s.sh
# Edited: 12/05/2013 03:28:08 PM EST
# Added UTCTIME	for 'tracking' 
# varnish-access.log in this format
# append /tmp/503s.out for better tracking.
UTCTIME=$(TZ=UTC date '+%d/%b/%Y:%H:%M:%S' --date="8 hours ago")
THISMINUTE=$(date '+%I:%M')
LASTMINUTE=$(date '+%I:%M' --date="1 minute ago")
COUNT=$(sed -n "/$LASTMINUTE/,/$THISMINUTE/p" /mnt/varnish/varnish-access.log | grep "HTTP/1.1\\" 503" | wc -l)
echo "$UTCTIME = $COUNT" >> /tmp/503s.out
#EOF

The UTC Time manipulation was necessary to make the output timestamp equivalent to the format used in the varnish-access.log

Code:

i.e.:
tail -1 /mnt/varnish/varnish-access.log | awk '{print $4}'
[05/Dec/2013:12:40:43

cat /tmp/503s.out
05/Dec/2013:12:32:01 = 0
05/Dec/2013:12:33:01 = 0
05/Dec/2013:12:34:01 = 0
05/Dec/2013:12:35:01 = 0
05/Dec/2013:12:36:01 = 0
05/Dec/2013:12:37:01 = 0
05/Dec/2013:12:38:01 = 0
05/Dec/2013:12:39:01 = 0
05/Dec/2013:12:40:01 = 0

http://www.bournetoraiseshell.com/ar...31202122038302