LinuxQuestions.org
Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 02-27-2012, 04:00 AM   #1
fernfrancis
Member
 
Registered: Feb 2009
Location: Goa(India)-Sharjah(UAE)
Distribution: RHEL,centos,fedora,ubuntu
Posts: 229

Rep: Reputation: 18
apache log analysis


HI All

we have setup a load balancing between our web servers ,earlier when we had a single server i used to use awstats to analyse the logs. now we have place the F5 device for load balancing

in apache the LOGFORMAT is specified as follows.
LogFormat "%{X-Forwarded-For}i| %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

A sample output from my logs is placed below

119.154.28.130| - - [27/Feb/2012:14:53:32 +0400] "GET /sites/all/themes/xyz/ajaxtabs/ajaxtabs.css HTTP/1.1" 200 1178 "http://www.xyz.com/home" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB7.2; .NET CLR 1.1.4322; InfoPath.2)"
92.96.21.246, 86.96.228.84| - - [27/Feb/2012:14:53:32 +0400] "POST /userfiles/bannerManagement/php/getAncillaryBannerDataUsr.php HTTP/1.1" 200 197 "http://www.xyz.com/userfiles/bannerManagement/banners/ancillaryBanner.swf?nocache=1330339964&siteURL=http://www.xyz.com/userfiles/bannerManagement/&id_section=43&countryVal=1&langVal=en" "Mozilla/5.0 (Windows NT 6.1; rv:10.0.2) Gecko/20100101 Firefox/10.0.2"
217.165.159.61, 86.96.226.22| - - [27/Feb/2012:14:53:32 +0400] "GET /userfiles/bannerManagement/upload/explorer/slides/SHJ_KWI_OB_banner_en_S3.jpg?nocache=undefined&uniq=133033297323013232.520854566246 HTTP/1.1" 200 27279 "-" "Mozilla/5.0 (Windows NT 6.0; rv:10.0.2) Gecko/20100101 Firefox/10.0.2"
122.173.49.127| - - [27/Feb/2012:14:53:32 +0400] "GET /sites/all/themes/xyz/images/booking_bg.gif HTTP/1.1" 200 3133 "http://www.xyz.com/home" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; InfoPath.2)"
122.161.14.75| - - [27/Feb/2012:14:53:32 +0400] "GET /sites/all/themes/xyz/images/text_bg.jpg HTTP/1.1" 200 638 "http://www.xyz.com/home-in" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/534.53.11 (KHTML, like Gecko) Version/5.1.3 Safari/534.53.10"
194.151.125.5| - - [27/Feb/2012:14:53:32 +0400] "GET /sites/all/themes/xyz/images/ancilliary_title_bg_new.gif HTTP/1.1" 200 3270 "http://www.xyz.com/home-nl?gclid=CNnG-rbkva4CFZAe3godrFnqPQ" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.1; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)"
94.43.199.198| - - [27/Feb/2012:14:53:32 +0400] "GET /scripts/js/smoothscroll.js HTTP/1.1" 200 2357 "http://www.xyz.com/faqs" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.6 (KHTML, like Gecko) Chrome/6.0.490.1 Safari/534.6"
117.205.18.221| - - [27/Feb/2012:14:53:32 +0400] "GET /userfiles/bannerManagement/templates/slideShow/css/style.css?nocache=undefined&uniq=133033297635916143.264813348651 HTTP/1.1" 200 4112 "-" "Mozilla/5.0 (Windows NT 5.1; rv:10.0.2) Gecko/20100101 Firefox/10.0.2"


Can i use awstats to analyse this , if yes please let me know, since when i analyse the log using awstats it takes the F5 virtual IP address and the number of connection seen are 2 since there are two virtual IP

please help out
 
Old 02-27-2012, 04:23 AM   #2
acid_kewpie
Moderator
 
Registered: Jun 2001
Location: UK
Distribution: Gentoo, RHEL, Fedora, Centos
Posts: 43,417

Rep: Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974
How can it be taking the LTM snat address when it's not in the logs there? You have two devices, so it's an HA pair? Are you not seeing HTTP monitor connections as these two addresses? On LTM you'll see the actual source IP of the connection to be the floating IP, not a self IP on the interface, so you'd only see one IP there.

Why have you modified the logformat other than for the XFF header? http://www.techstacks.com/howto/log-...in-apache.html
 
Old 02-27-2012, 10:19 PM   #3
allinurl
LQ Newbie
 
Registered: Mar 2009
Posts: 6

Rep: Reputation: 0
Thumbs up

I'll definitely use GoAccess (http://goaccess.prosoftcorp.com/), it parses extremely fast and pretty easy to use. I tested your sample log and it worked fine.
 
Old 02-28-2012, 03:58 AM   #4
fernfrancis
Member
 
Registered: Feb 2009
Location: Goa(India)-Sharjah(UAE)
Distribution: RHEL,centos,fedora,ubuntu
Posts: 229

Original Poster
Rep: Reputation: 18
can some one give me a command to seperate the IP address and the rest

for example
119.154.28.130| - - [27/Feb/2012:14:53:32 +0400] "GET /sites/all/themes/xyz/ajaxtabs/ajaxtabs.css HTTP/1.1" 200 1178 "http://www.xyz.com/home" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB7.2; .NET CLR 1.1.4322; InfoPath.2)"

currently in some places i get

92.96.21.246, 86.96.228.84| - - [27/Feb/2012:14:53:32 +0400] "POST /userfiles/bannerManagement/php/getAncillaryBannerDataUsr.php HTTP/1.1" 200 197 "http://www.xyz.com/userfiles/bannerManagement/banners/ancillaryBanner.swf?nocache=1330339964&siteURL=http://www.xyz.com/userfiles/bannerManagement/&id_section=43&countryVal=1&langVal=en" "Mozilla/5.0 (Windows NT 6.1; rv:10.0.2) Gecko/20100101 Firefox/10.0.2"


here i need to get the client IP

i used something like this
tail -f /var/log/httpd/access.20120228.log | awk -F ',' '{ print $1" "$NF}'


but it doesnt work. if someone can give me a appropriate filter

Please help me with that so that i can pass it though the awstats software

Last edited by fernfrancis; 02-28-2012 at 04:28 AM.
 
Old 02-28-2012, 05:11 AM   #5
acid_kewpie
Moderator
 
Registered: Jun 2001
Location: UK
Distribution: Gentoo, RHEL, Fedora, Centos
Posts: 43,417

Rep: Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974Reputation: 1974
any chance you could actually acknowledge our responses instead of ignoring them?
 
Old 02-28-2012, 11:42 PM   #6
fernfrancis
Member
 
Registered: Feb 2009
Location: Goa(India)-Sharjah(UAE)
Distribution: RHEL,centos,fedora,ubuntu
Posts: 229

Original Poster
Rep: Reputation: 18
HI acid_kewpie

i am not ignoring your response its just that it doesnt meet my requirement.

If you could help me with what i exactly need would be of great help
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Apache Log Analysis Program nick1 Linux - Software 1 03-17-2006 02:56 PM
Lire (log analysis, log report) no report in Mandriva 2005 LE (desktop usage) Emmanuel_uk Mandriva 0 01-16-2006 03:11 AM
Apache Log Analysis dkpw Slackware 7 06-04-2005 09:48 AM
apache log analysis? belorion Linux - Networking 1 05-12-2005 12:52 PM
Analysis Apache log treotan Linux - Networking 1 07-09-2004 11:29 PM


All times are GMT -5. The time now is 12:21 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration