LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 05-21-2015, 11:53 PM   #1
mfoley
Senior Member
 
Registered: Oct 2008
Posts: 1,098

Rep: Reputation: 100Reputation: 100
Need to exclude certain messages from Apache access_log


I've tried this question on other forums and nobody's responded - hopefully I'll have better luck here!

I am running Apache 2.2.9 and Tomcat 6.0.14 on Slackware 13.37.0, kernel 2.6.37.6-smp.

I am logging messages to /usr/local/apache2/logs/access_log as follows:

Code:
98.102.63.107 - - [21/May/2015:10:51:56 -0400] "GET /ohprs/images/memoriam/JackHolland.jpg HTTP/1.1" 200 10502
98.102.63.107 - - [21/May/2015:10:51:56 -0400] "GET /ohprs/images/memoriam/AndrewBaldridge.jpg HTTP/1.1" 200 9147
98.102.63.107 - - [21/May/2015:10:51:58 -0400] "GET /ohprs/images/stripes.png HTTP/1.1" 304 -
98.102.63.107 - - [21/May/2015:10:51:58 -0400] "GET /ohprs/images/logo.png HTTP/1.1" 304 -
I want to eliminate logging of any accesses to the ohprs/images folder. In my httpd.conf file I have:

SetEnvIf Request_URI "/ohprs/images" dontlog
CustomLog "logs/access_log" combined env=!dontlog

I've tried various combinations including using "Referer" as the attribute and with or without the leading "/" in the regex. Nothing seems to work.

Does this version of Apache support SetEnvIf? The docs seem to indicate it does. Do I need to load some module? Perhaps I have the SetEnvIf directive in the wrong place in httpd.conf? Any ideas?

Here's my httpd.conf:
Code:
ServerRoot "/usr/local/apache2"
PidFile /var/run/apache.pid
Listen 80
LoadModule    jk_module  lib/mod_jk.so
JkWorkersFile conf/workers.properties
JkShmFile     /var/log/httpd/mod_jk.shm
JkLogFile     /var/log/httpd/mod_jk.log
JkLogLevel    info
JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
JkMount  /ohprs/* worker1
JkMount  /ohprs worker1
JkMount  /ohprs2/* worker1
JkMount  /ohprs2 worker1
JkMount  /office/* worker1
JkMount  /office worker1

<IfModule !mpm_netware_module>
<IfModule !mpm_winnt_module>
User apache
Group apache

</IfModule>
</IfModule>

ServerAdmin mfoley@novatec-inc.com
ServerName www.ohprs.org
DocumentRoot "/usr/local/apache2/htdocs"

Rewritelog logs/rewrite.log
Rewriteloglevel 5

<Directory />
    Options FollowSymLinks
    AllowOverride None
    Order deny,allow
    Deny from all
</Directory>

<Directory "/usr/local/apache2/htdocs">
    Options Indexes FollowSymLinks ExecCGI

    AllowOverride All

    Order allow,deny
    Allow from all

</Directory>

<IfModule dir_module>
    DirectoryIndex index.html
</IfModule>

<FilesMatch "^\.ht">
    Order allow,deny
    Deny from all
    Satisfy All
</FilesMatch>

<Location /office>
RewriteEngine On
RewriteCond %{HTTPS} !on
RewriteRule ^/?(.*) https://%{SERVER_NAME}%{REQUEST_URI} [L,R=301]
</Location>

<Location /ohprs>
RewriteEngine On
RewriteCond %{HTTPS} !on
RewriteRule ^/?(.*) https://%{SERVER_NAME}%{REQUEST_URI} [L,r=301]
</Location>

<Location /ohprs2>
RewriteEngine On
RewriteCond %{HTTPS} !on
RewriteRule ^/?(.*) https://%{SERVER_NAME}%{REQUEST_URI} [L,r=301]
</Location>

<Location /ohprs/upload>
    AuthType Basic
    AuthName "Restricted"
    AuthUserFile conf/passwords
    Require valid-user
    AuthBasicProvider file
</Location>

  CheckCaseOnly on

ErrorLog "logs/error_log"

LogLevel warn

<IfModule log_config_module>
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common

    <IfModule logio_module>
      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
    </IfModule>

    SetEnvIf Request_URI "/ohprs/images" dontlog
    CustomLog "logs/access_log" combined env=!dontlog

</IfModule>

<IfModule alias_module>

    ScriptAlias /cgi-bin/ "/usr/local/apache2/cgi-bin/"

</IfModule>

<IfModule cgid_module>
</IfModule>

<Directory "/usr/local/apache2/cgi-bin">
    AllowOverride None
    Options None
    Order allow,deny
    Allow from all
</Directory>

DefaultType text/plain

<IfModule mime_module>
    TypesConfig conf/mime.types

    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz

    AddHandler cgi-script .cgi

</IfModule>

Include conf/extra/httpd-ssl.conf
<IfModule ssl_module>
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin
</IfModule>
 
Old 05-22-2015, 03:17 AM   #2
bathory
LQ Guru
 
Registered: Jun 2004
Location: Piraeus
Distribution: Slackware
Posts: 11,882

Rep: Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608
Hi,

You should use:
Code:
SetEnvIf Request_URI "^/ohprs/images/" dontlog
Regards
 
Old 05-26-2015, 08:15 PM   #3
mfoley
Senior Member
 
Registered: Oct 2008
Posts: 1,098

Original Poster
Rep: Reputation: 100Reputation: 100
Tried that. Added the following to my httpd.conf -- also did this on a newer Slackware64 14.1, kernel 3.10.7, Apache version 2.4.10.

LoadModule setenvif_module lib64/httpd/modules/mod_setenvif.so
:
:
LogFormat "h %l %u %t "%r" %>s %b" common
SetEnvIf Request_URI "^/calendars" dontlog
CustomLog "/var/log/httpd/access_log" common=!dontlog

access_log:
Code:
192.168.0.56 - - [26/May/2015:20:05:23 -0400] "GET /calendars/Smith_Calendar.ics HTTP/1.1" 304 -
192.168.0.56 - - [26/May/2015:20:05:23 -0400] "GET /calendars/Mike_Calendar.ics HTTP/1.1" 304 -
192.168.0.56 - - [26/May/2015:20:05:23 -0400] "GET /calendars/Mark_Calendar.ics HTTP/1.1" 304 -
Any other ideas?
 
Old 05-27-2015, 01:49 AM   #4
bathory
LQ Guru
 
Registered: Jun 2004
Location: Piraeus
Distribution: Slackware
Posts: 11,882

Rep: Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608
Quote:
LogFormat "h %l %u %t "%r" %>s %b" common
SetEnvIf Request_URI "^/calendars" dontlog
CustomLog "/var/log/httpd/access_log" common=!dontlog
You have some errors (maybe typos). The correct is:
Code:
LogFormat "%h %l %u %t "%r" %>s %b" common
SetEnvIf Request_URI "^/calendars" dontlog
CustomLog "/var/log/httpd/access_log" common env=!dontlog
 
Old 05-27-2015, 10:45 AM   #5
mfoley
Senior Member
 
Registered: Oct 2008
Posts: 1,098

Original Poster
Rep: Reputation: 100Reputation: 100
Yes, I saw that later and fixed it I now have:

SetEnvIf Request_URI "^/calendars" dontlog=true
CustomLog "/var/log/httpd/access_log" common env=!dontlog

but it still doesn't work. The following link indicates that this does not work if redirecting to SSL: http://www.apachelounge.com/viewtopic.php?p=22266, which I am doing. The poster in that link says his SetEnvIf did not work until he turned off the SSL redirection. Otherwise, no solution was posted. I am not in a position to turn this off and for production use don't want to. Not really sure why this wouldn't work redirecting to SSL. Any ideas?

I have the following in my DOCUMENT_ROOT/.htaccess:

RewriteEngine On
RewriteCond {HTTPS} !^on
RewriteRule .* https://%{SERVER_NAME}%{REQUEST_URI} [L,R]
 
Old 05-27-2015, 10:56 AM   #6
mfoley
Senior Member
 
Registered: Oct 2008
Posts: 1,098

Original Poster
Rep: Reputation: 100Reputation: 100
More info: I turned off the http rewrite. If I reference the /calendars directory as http://hostname.com/calendars, then the nolog function works. However, even with the rewrite off, if I use https://hostname.com/calendars the request gets logged to access_log. So, using https at all causes the SetEnvIf to fail.

Why? How to fix?
 
Old 05-27-2015, 02:02 PM   #7
bathory
LQ Guru
 
Registered: Jun 2004
Location: Piraeus
Distribution: Slackware
Posts: 11,882

Rep: Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608
Quote:
Originally Posted by mfoley View Post
More info: I turned off the http rewrite. If I reference the /calendars directory as http://hostname.com/calendars, then the nolog function works. However, even with the rewrite off, if I use https://hostname.com/calendars the request gets logged to access_log. So, using https at all causes the SetEnvIf to fail.

Why? How to fix?
Sorry don't know how to preserve the env. variable when using mod_rewrire
If it's acceptable, you can use the same technique inside the https vhost stanza and just use a different file to log ssl access, e.g:
Code:
SetEnvIf Request_URI "^/calendars" dontlog
CustomLog /var/log/httpd/ssl_access_log common env=!dontlog
 
Old 05-27-2015, 10:42 PM   #8
mfoley
Senior Member
 
Registered: Oct 2008
Posts: 1,098

Original Poster
Rep: Reputation: 100Reputation: 100
Worth trying. I have the following in my httpd-ssl.conf:

CustomLog "/var/log/httpd/ssl_request_log" \
"t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x "%r" %b"

I've modified/added to give:

SetEnvIf Request_URI "^/calendars" dontlog
CustomLog "/var/log/httpd/ssl_request_log" \
"%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x "%r" %b" env=!dontlog

I'll post back results.
 
Old 05-28-2015, 01:45 AM   #9
mfoley
Senior Member
 
Registered: Oct 2008
Posts: 1,098

Original Poster
Rep: Reputation: 100Reputation: 100
The SetEnvIf I put into the httpd-ssl.conf worked. No "/calendars" URIs are logged to ssl_request_log. However, these URIs are still logged to access_log. Still at a loss as to how to not log things in access_log. Seems like an Apache bug!

How about if I log NOTHING to access_log in the main httpd.conf and direct all ssl logging to the access_log instead of ssl_request_log?

How do I turn off non-ssl logging? I've tried commenting out the CustomLog line in httpd.conf, but it's still logging!

Last edited by mfoley; 05-28-2015 at 01:55 AM.
 
Old 05-28-2015, 03:10 AM   #10
bathory
LQ Guru
 
Registered: Jun 2004
Location: Piraeus
Distribution: Slackware
Posts: 11,882

Rep: Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608
Quote:
How about if I log NOTHING to access_log in the main httpd.conf and direct all ssl logging to the access_log instead of ssl_request_log?

How do I turn off non-ssl logging? I've tried commenting out the CustomLog line in httpd.conf, but it's still logging!
If you don't want logging of http, use /dev/null:
Code:
CustomLog /dev/null common
 
Old 05-29-2015, 12:25 AM   #11
mfoley
Senior Member
 
Registered: Oct 2008
Posts: 1,098

Original Poster
Rep: Reputation: 100Reputation: 100
This thing's like a zombie cockroach! I can't kill it! I've commented out the original CustomLog directives in httpd.conf and replaced with your suggested "CustomLog /dev/null common", yet I'm *still* getting log messages in access_log!!!! I've checked all the files included by httpd.conf and none other than the httpd-ssl.conf have a CustomLog directive. I've even turned off (commented out) the CustomLog in httpd-ssl.conf -- but they keep coming! I've shutdown httpd for several minutes, verified no httpd task are running, then started. Really, how do I stop logging?

Just to prove I'm not insane (in this instance) here's my httpd.conf, excluding the many LogModules:
Code:
ServerRoot "/usr"

Listen 80

LoadModule authn_file_module lib64/httpd/modules/mod_authn_file.so
:
many more
:
:

RewriteEngine On
RewriteRule ^/DVR$ DVR/ [R]

ProxyPass /DVR/ http://192.168.0.24/

<IfModule unixd_module>
User apache
Group apache

</IfModule>

ServerAdmin webadmin@ohprs.org

ServerName mail.ohprs.org:80

<Directory />
    AllowOverride none
    Require all denied
</Directory>

DocumentRoot "/srv/httpd/htdocs"
<Directory "/srv/httpd/htdocs">
    Options FollowSymLinks

    AllowOverride All

    Require all granted
</Directory>

<IfModule dir_module>
    DirectoryIndex index.html index.php
</IfModule>

<Files ".ht*">
    Require all denied
</Files>

ErrorLog "/var/log/httpd/error_log"

LogLevel warn

<IfModule log_config_module>
    LogFormat "h l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common

    <IfModule logio_module>
      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
    </IfModule>

CustomLog /dev/null common

</IfModule>

<IfModule alias_module>

    ScriptAlias /cgi-bin/ "/srv/httpd/cgi-bin/"

</IfModule>

<IfModule cgid_module>
</IfModule>

<Directory "/srv/httpd/cgi-bin">
    AllowOverride None
    Options None
    Require all granted
</Directory>

<IfModule mime_module>
    TypesConfig /etc/httpd/mime.types

    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz

</IfModule>

Include /etc/httpd/extra/httpd-dav.conf

<IfModule proxy_html_module>
Include /etc/httpd/extra/proxy-html.conf
</IfModule>

Include /etc/httpd/extra/httpd-ssl.conf
<IfModule ssl_module>
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin
</IfModule>

Include /etc/httpd/mod_php.conf

Last edited by mfoley; 05-29-2015 at 12:27 AM.
 
Old 05-29-2015, 03:48 AM   #12
bathory
LQ Guru
 
Registered: Jun 2004
Location: Piraeus
Distribution: Slackware
Posts: 11,882

Rep: Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608
Quote:
This thing's like a zombie cockroach! I can't kill it! I've commented out the original CustomLog directives in httpd.conf and replaced with your suggested "CustomLog /dev/null common", yet I'm *still* getting log messages in access_log!!!! I've checked all the files included by httpd.conf and none other than the httpd-ssl.conf have a CustomLog directive. I've even turned off (commented out) the CustomLog in httpd-ssl.conf -- but they keep coming! I've shutdown httpd for several minutes, verified no httpd task are running, then started. Really, how do I stop logging?
It should work.
Could be that you're using "TransferLog" instead of "CustomLog" somewhere in your configs.
Or CustomLog is defined in an included file for vhosts setup. You can run a recursive find under the apache config directory, to see where it's used:
Code:
find /etc/httpd -type f -exec grep -i Customlog {} /dev/null \;
 
Old 05-29-2015, 07:31 PM   #13
mfoley
Senior Member
 
Registered: Oct 2008
Posts: 1,098

Original Poster
Rep: Reputation: 100Reputation: 100
Yep, you nailed it! There was a TransferLog directive in httpd-ssl.conf. When I commented that out all logging stopped. So, I had the following in the httpd-ssl.conf <Vitural Host> section:

TransferLog "/var/log/httpd/access_log"
CustomLog "/var/log/httpd/ssl_request_log" "t h {SSL_PROTOCOL}x %{SSL_CIPHER}x "%r" %b"

and the following in httpd.conf in the <IfModule log_config_module> section:

LogFormat "%h %l %u %t "%r" %>s %b" common
CustomLog "/var/log/httpd/access_log" common

Seems like an awful lot of logging going on. With respect to logging, this was the default setup. I would have expected with the TransferLog (httpd-ssl.conf) and the CustomLog (httpd.conf) both specified to have gotten 2 messages per access in access_log, but I didn't. Any idea why?

I've commented out the "CustomLog /dev/null common" in httpd.conf and the TransferLog in httpd-ssl.conf and now have only the following in httpd-ssl.conf:

SetEnvIf Request_URI "^/calendars" dontlog
CustomLog "/var/log/httpd/access_log" \
"%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x "%r" %b" env=!dontlog

I'll let this run a bit, then test re-enabling the CustomLog in the main httpd.conf. I'm thinking that when I did the SetEnvIf there the TransferLog "took over" and logged the message anyway.

Last edited by mfoley; 05-29-2015 at 07:34 PM.
 
Old 05-30-2015, 03:39 AM   #14
bathory
LQ Guru
 
Registered: Jun 2004
Location: Piraeus
Distribution: Slackware
Posts: 11,882

Rep: Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608Reputation: 1608
Quote:
I would have expected with the TransferLog (httpd-ssl.conf) and the CustomLog (httpd.conf) both specified to have gotten 2 messages per access in access_log, but I didn't. Any idea why?
You didn't because TransferLog specifies the logging for the https server, while CustomLog (httpd.conf) does the same for the http.
If you leave both TransferLog and CustomLog enabled in the https section (the default), you'll get 2 logs for the same request. The usual from TransferLog (written in access_log) and one containing also the ssl protocol from CustomLog (written in ssl_request_log).


Quote:
I'm thinking that when I did the SetEnvIf there the TransferLog "took over" and logged the message anyway.
Nah. TransferLog was logging the request when using https regardless the SetEnvIf in the http section
 
Old 06-02-2015, 12:10 PM   #15
mfoley
Senior Member
 
Registered: Oct 2008
Posts: 1,098

Original Poster
Rep: Reputation: 100Reputation: 100
bathory: "If you leave both TransferLog and CustomLog enabled in the https section (the default), you'll get 2 logs for the same request."

Well, when I did have both enabled I didn't see 2 messages. But no matter, I believe I've solved my problem by comments out ALL logging in httpd-ssl.conf and having only the following in httpd.conf:

LogLevel warn
LogFormat "h %l %u %t "%r" %>s %b" common
SetEnvIf Request_URI "^/calendars" dontlog=true
CustomLog "/var/log/httpd/access_log" common env=!dontlog

That kills off requests for the calendars folder.

I believe this issue is resolved, though confusing. Thanks for your help.
 
  


Reply

Tags
httpdconf, setenv


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
How to read the access_log of Apache? Kayaker Slackware 12 06-02-2009 02:22 PM
apache access_log questions shilo Linux - Software 9 06-07-2004 03:40 PM
apache access_log woes thew00t Linux - Software 1 02-23-2004 08:26 AM
apache access_log permissions mirage_3d Linux - Networking 2 12-06-2003 09:54 PM
Apache access_log question WorldBuilder Linux - Networking 7 11-01-2003 07:05 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 02:20 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration