How to read the access_log of Apache?

Kayaker · 04-06-2003, 09:48 AM

Folks,

This is really an apache question - not slackware question. But I have always received excellent advice from Slackware folks than anyone else. I am taking advantage of knowledge of slack floks once again.

A standard apache installation yields an access log file called
/usr/local/apache2/logs/access_log
and after the server is up running for a while, it is usually loaded with contents like:

65.33.94.190 - - [05/Apr/2003:17:26:27 -0500] "GET /scripts/root.exe?/c+dir HTTP/1.0" 404 276

65.33.94.190 - - [05/Apr/2003:17:26:28 -0500] "GET /MSADC/root.exe?/c+dir HTTP/1.0" 404 274

65.33.94.190 - - [05/Apr/2003:17:26:29 -0500] "GET /c/winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 284

65.33.94.190 - - [05/Apr/2003:17:26:41 -0500] "GET /scripts/..%%35c../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 400 288

12.246.123.60 - - [04/Apr/2003:13:01:15 -0500] "GET /scripts/root.exe?/c+dir HTTP/1.0" 404 276

80.133.155.17 - - [03/Apr/2003:10:35:54 -0500] "PROPFIND /admin%24 HTTP/1.1" 405 305

What are these stuff?

How do I read the access log.

I would appreciate the answers to my question come in the following formats:

1) a simplified answer such as
the_meaning_of_the_1st_field
the_meaning_of_the_2nd_field
the_meaning_of_the_3rd_field
the_meaning_of_the_4th_field
the_meaning_of_the_5th_field
the_meaning_of_the_6th_field
....

or

2) a pointer (a hyperlink) to the documenation where a detailed explanation can be found.

By the way, does access_log provide any indication of security breach of the web server? If so, how does one tell by reading the log?

Many Thanks.

Kayaker

david_ross · 04-06-2003, 10:30 AM

Take a look at
http://www.serverwatch.com/tutorials...le.php/1127521

I once wrote a script to show the error log (I use it a lot when debugging programs) You could probably adapt it to show the access log in a mor ereadable format too:

Code:

#!/usr/bin/perl
if(!$ENV{'QUERY_STRING'}){$do="15"}
else{$do=$ENV{'QUERY_STRING'}}
print "Content-type: text/html\n\n<HTML><TITLE>View Log</TITLE><BODY><H2>LOGS</H2><TABLE border=1>";
open(LOG, "/var/log/httpd/error_log") || die("Could not open log");
@line = <LOG>;
close(LOG);
print $line[1];
$num = @line;
for($done=0;$done<$do;$done++){
$num--;
if($line[$num] =~ /^\[.*:..:.*\]/){
$line[$num] =~ tr/\[/ /;
($time, $typ, $cli, $info) = split (/\]/, $line[$num]);
}
else{$info=$line[$num]}
if($line[$num] =~ /\(2\)/){
$cli = "";
$info=$cli
}
if($tm ne $time){print "<TR><TD colspan=4><HR></TD></TR>"}
print "<TR><TD width=100>$time</TD><TD>$typ</TD><TD width=50>$cli</TD><TD>$info</TD></TR>";
$tm=$time;
}
print "</TABLE></BODY></HTML>";
exit;

Kayaker · 04-06-2003, 12:07 PM

Thank you David.

I found this page after posting the question.

http://httpd.apache.org/docs/logs.html#accesslog

perhaps it will be of interest to others as well.

Kayaker

MasterC · 04-06-2003, 12:44 PM

Quote:

Originally posted by david_ross
Take a look at
http://www.serverwatch.com/tutorials...le.php/1127521

I once wrote a script to show the error log (I use it a lot when debugging programs) You could probably adapt it to show the access log in a mor ereadable format too:

Code:

#!/usr/bin/perl if(!$ENV{'QUERY_STRING'}){$do="15"} else{$do=$ENV{'QUERY_STRING'}} print "Content-type: text/html\n\n<HTML><TITLE>View Log</TITLE><BODY><H2>LOGS</H2><TABLE border=1>"; open(LOG, "/var/log/httpd/error_log") || die("Could not open log"); @line = <LOG>; close(LOG); print $line[1]; $num = @line; for($done=0;$done<$do;$done++){ $num--; if($line[$num] =~ /^\[.*:..:.*\]/){ $line[$num] =~ tr/\[/ /; ($time, $typ, $cli, $info) = split (/\]/, $line[$num]); } else{$info=$line[$num]} if($line[$num] =~ /$2$/){ $cli = ""; $info=$cli } if($tm ne $time){print "<TR><TD colspan=4><HR></TD></TR>"} print "<TR><TD width=100>$time</TD><TD>$typ</TD><TD width=50>$cli</TD><TD>$info</TD></TR>"; $tm=$time; } print "</TABLE></BODY></HTML>"; exit;

Where would I put this, and how would I use it (and what does it do

)?

Looks cool though

Cool

david_ross · 04-06-2003, 01:03 PM

I assume you are talking about the script.
1) Put it in a file called log.cgi within your cgi-bin
2) Give it executeable permissions
3) Make sure the error log file is pointing to the right place
4) Set the permissions on the logs/logs directory so the web server can read it.
5) Access http://yourhost/cgi-bin/log.cgi

That's it! It automatically displays the last 15 lines of the error log in an easy to read table. It tries to group errors related to each individual access and puts an <HR> (horizontal rule) between them.

If you want to see more than the last 15 lines then append the number of lines to view to the filename as a query string. To disply the last 100 - http://yourhost/cgi-bin/log.cgi?100

Let me know if you have any problems.

MasterC · 04-06-2003, 01:33 PM

Woohoo! Worked great, thanks!

I tried to make an access log the same way, but it didn't quite work out the same, the date/time wasn't displayed, and all entries were grouped into one. But I will work on that...

Thanks!

Cool

MasterC · 04-06-2003, 01:36 PM

Unlrelated:

You should sign up for affero. I was going to "affero" you, but it looks like you aren't signed up. Here's a thread with some info on it:
http://www.linuxquestions.org/questi...threadid=25730

And here's my search string:
http://www.linuxquestions.org/questi...der=descending

For even more info

Cool

david_ross · 04-06-2003, 01:40 PM

I keep meaning to - I just never get round to it. No time like the present I suppose!

I'm glad you like it. I was just getting annoyed at reading horrible raw logs. There are plenty of analysers for access logs but nothing for us poor developers. There are a few simple enhancements that I may make at some point but consider it as GPL

MasterC · 04-06-2003, 03:15 PM

Cool, thanks again, and can you provide me any tips on how to go about setting this up for other logs, such as access_log?

david_ross · 04-06-2003, 03:17 PM

Since you just clicked the Affreo - give me 10 mins!

david_ross · 04-06-2003, 05:17 PM

The longest 10 minutes of my life!

Code:

#!/usr/bin/perl

# Start Config

$access_log = "/var/log/httpd/access_log";
$error_log = "/var/log/httpd/error_log";
$default_log = "error"; # access or error
$default_lines = "15";

# End Config

# Print the top of the page
print <<"EOF";
Content-type: text/html

<HTML>
<HEAD>
<TITLE>Viewing $in{'log'} log</TITLE>
</HEAD>
<BODY>
<H1>Log viewing options</H1>
<FORM method="get">
<TABLE>
<TR><TH>Log Type:</TH><TD>Error<INPUT type="radio" value="error" name="log">
&nbsp;&nbsp;Access<INPUT type="radio" value="access" name="log"></TD></TR>
<TR><TH>Lines to show:</TH><TD><INPUT type="text" size="5" name="lines" value="$default_lines"></TD></TR>
<TR><TH>&nbsp</TH><TD><INPUT type="submit" value="Show Log"></TD></TR>
</TABLE>
</FORM>
EOF

# Get values from the query string (not many so we'll not use post)
foreach $pair (split(/&/, $ENV{'QUERY_STRING'})){
$pair =~ tr/+/ /;
($name, $value) = split(/=/, $pair);
$name =~ s/%(..)/pack("C", hex($1))/eg;
$value =~ s/%(..)/pack("C", hex($1))/eg;
$in{$name} = $value;
}
# If specific options were not given then use the defaults
if(!$in{'log'} && ($in{'log'} ne "access" || $in{log} ne "error")){$in{'log'} = $default_log}
if(!$in{'lines'}){$in{'lines'} = $default_lines}
&showlog;

print "</BODY></HTML>";

sub showlog{
print "<H1>Viewing $in{'log'} log</H1><TABLE border=1 width=90%>";
if($in{'log'} eq "access"){
open(LOG, "$access_log");
@line = <LOG>;
close(LOG);
$num = @line;
for($done=0;$done<$in{'lines'};$done++){
$num--;
($start, $request, $response, $ref, $other, $browser, $end) = split (/\"/, $line[$num]);
($browser, $end) = split (/ \(/, $browser);
($client, $other) = split (/\s/, $line[$num]);
($start, $end) = split (/\- /, $line[$num]);
($user, $bad) = split (/ \-/, $end);
($start, $end) = split (/\[/, $line[$num]);
($time, $line[$num]) = split (/\]/, $end);
if($response =~ /^ 200/){$resp = "OK!"}
if($response =~ /^ 500/){$resp = "Server error"}
if($response =~ /^ 404/){$resp = "Page not found"}
if($response =~ /^ 403/){$resp = "Authorisation required"}
if($response =~ /^ 401/){$resp = "Forbidden"}
if($response =~ /^ 400/){$resp = "Bad request"}
if($tm ne $time){print "<TR><TD colspan=4><HR></TD></TR>"}
print "<TR><TD width=35%><B>At:</B> $time<BR><B>User:</B> $client $user<BR><B>Browser:</B> $browser</TD>";
print "<TD width=65%><B>From page:</B> $ref<BR><B>Request:</B> $request<BR><B>Response:</B> $resp ($response)</TD></TR>";
$tm=$time;
}
}
elsif($in{'log'} eq "error"){
open(LOG, "$error_log");
@line = <LOG>;
close(LOG);
$num = @line;
for($done=0;$done<$in{'lines'};$done++){
$num--;
if($line[$num] =~ /^\[.*:..:.*\]/){
$line[$num] =~ tr/\[/ /;
($time, $typ, $cli, $info) = split (/\]/, $line[$num]);
}
else{$info=$line[$num]}
if($line[$num] =~ /\(2\)/){
$cli = "";
$info=$cli;
}
if($tm ne $time){print "<TR><TD colspan=4><HR></TD></TR>"}
print "<TR><TD width=100>$time</TD><TD>$typ</TD><TD width=50>$cli</TD><TD>$info</TD></TR>";
$tm=$time;
}
}
print "</TABLE>";
}

exit;

During which I almost completly rewrote a script then deleted half of it to find that for some reason my editor wouldn't let me undo. The I rewrote that stuff again, added bits - decided how to display the access log and tested it.

Oh I almost forgot the part where I tried to be clever and sent my server running 2 permenant loops (don't worry I fixed that :D)

Anyway I think I'll stop for the night before I delete anything else or accidentally advise soemone to run "rm -rf /"

Let me know how you get on with it - there are now some config options at the top that you may want to set first. if you have any problems or think I made a boo boo then I'll see what I can do.

MasterC · 04-06-2003, 06:40 PM

Wow! That works great! Thanks for all that work, really appreciate it

Cool

ersimpson75 · 06-02-2009, 01:22 PM

Hi David,
I tried the first script and it works nicely!
What I would like to know is how to change it so that it will look for a log file that changes the name from day to day.
I tried several wild card substitutions but with no success.
Thanks, Ed Simpson