apache log analysis
 

HI All

we have setup a load balancing between our web servers ,earlier when we had a single server i used to use awstats to analyse the logs. now we have place the F5 device for load balancing

in apache the LOGFORMAT is specified as follows.
LogFormat "%{X-Forwarded-For}i| %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

A sample output from my logs is placed below

119.154.28.130| - - [27/Feb/2012:14:53:32 +0400] "GET /sites/all/themes/xyz/ajaxtabs/ajaxtabs.css HTTP/1.1" 200 1178 "http://www.xyz.com/home" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB7.2; .NET CLR 1.1.4322; InfoPath.2)"
92.96.21.246, 86.96.228.84| - - [27/Feb/2012:14:53:32 +0400] "POST /userfiles/bannerManagement/php/getAncillaryBannerDataUsr.php HTTP/1.1" 200 197 "http://www.xyz.com/userfiles/bannerManagement/banners/ancillaryBanner.swf?nocache=1330339964&siteURL=http://www.xyz.com/userfiles/bannerManagement/&id_section=43&countryVal=1&langVal=en" "Mozilla/5.0 (Windows NT 6.1; rv:10.0.2) Gecko/20100101 Firefox/10.0.2"
217.165.159.61, 86.96.226.22| - - [27/Feb/2012:14:53:32 +0400] "GET /userfiles/bannerManagement/upload/explorer/slides/SHJ_KWI_OB_banner_en_S3.jpg?nocache=undefined&uniq=133033297323013232.520854566246 HTTP/1.1" 200 27279 "-" "Mozilla/5.0 (Windows NT 6.0; rv:10.0.2) Gecko/20100101 Firefox/10.0.2"
122.173.49.127| - - [27/Feb/2012:14:53:32 +0400] "GET /sites/all/themes/xyz/images/booking_bg.gif HTTP/1.1" 200 3133 "http://www.xyz.com/home" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; InfoPath.2)"
122.161.14.75| - - [27/Feb/2012:14:53:32 +0400] "GET /sites/all/themes/xyz/images/text_bg.jpg HTTP/1.1" 200 638 "http://www.xyz.com/home-in" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/534.53.11 (KHTML, like Gecko) Version/5.1.3 Safari/534.53.10"
194.151.125.5| - - [27/Feb/2012:14:53:32 +0400] "GET /sites/all/themes/xyz/images/ancilliary_title_bg_new.gif HTTP/1.1" 200 3270 "http://www.xyz.com/home-nl?gclid=CNnG-rbkva4CFZAe3godrFnqPQ" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.1; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)"
94.43.199.198| - - [27/Feb/2012:14:53:32 +0400] "GET /scripts/js/smoothscroll.js HTTP/1.1" 200 2357 "http://www.xyz.com/faqs" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.6 (KHTML, like Gecko) Chrome/6.0.490.1 Safari/534.6"
117.205.18.221| - - [27/Feb/2012:14:53:32 +0400] "GET /userfiles/bannerManagement/templates/slideShow/css/style.css?nocache=undefined&uniq=133033297635916143.264813348651 HTTP/1.1" 200 4112 "-" "Mozilla/5.0 (Windows NT 5.1; rv:10.0.2) Gecko/20100101 Firefox/10.0.2"


Can i use awstats to analyse this , if yes please let me know, since when i analyse the log using awstats it takes the F5 virtual IP address and the number of connection seen are 2 since there are two virtual IP

please help out

How can it be taking the LTM snat address when it's not in the logs there? You have two devices, so it's an HA pair? Are you not seeing HTTP monitor connections as these two addresses? On LTM you'll see the actual source IP of the connection to be the floating IP, not a self IP on the interface, so you'd only see one IP there.

Why have you modified the logformat other than for the XFF header? http://www.techstacks.com/howto/log-...in-apache.html

I'll definitely use GoAccess (http://goaccess.prosoftcorp.com/), it parses extremely fast and pretty easy to use. I tested your sample log and it worked fine.

can some one give me a command to seperate the IP address and the rest

for example
119.154.28.130| - - [27/Feb/2012:14:53:32 +0400] "GET /sites/all/themes/xyz/ajaxtabs/ajaxtabs.css HTTP/1.1" 200 1178 "http://www.xyz.com/home" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB7.2; .NET CLR 1.1.4322; InfoPath.2)"

currently in some places i get

92.96.21.246, 86.96.228.84| - - [27/Feb/2012:14:53:32 +0400] "POST /userfiles/bannerManagement/php/getAncillaryBannerDataUsr.php HTTP/1.1" 200 197 "http://www.xyz.com/userfiles/bannerManagement/banners/ancillaryBanner.swf?nocache=1330339964&siteURL=http://www.xyz.com/userfiles/bannerManagement/&id_section=43&countryVal=1&langVal=en" "Mozilla/5.0 (Windows NT 6.1; rv:10.0.2) Gecko/20100101 Firefox/10.0.2"


here i need to get the client IP

i used something like this
tail -f /var/log/httpd/access.20120228.log | awk -F ',' '{ print $1" "$NF}'


but it doesnt work. if someone can give me a appropriate filter

Please help me with that so that i can pass it though the awstats software

any chance you could actually acknowledge our responses instead of ignoring them?

HI acid_kewpie

i am not ignoring your response its just that it doesnt meet my requirement.

If you could help me with what i exactly need would be of great help