how to i clear all my apache log files?

steve51184 · 11-14-2010, 08:18 PM

Hey guys i've had my dedicated server for about 16 months now and my log files are huge and I host a few small sites but one of them is quite large and can get a log file (access.log and error.log) of 1-2gb a month so over a year can take up almost 30gb just for that site so I clean them out regularly but I'm wanting a more automated way to do this

i've been thinking of using the find command using something like this but it isn't working

Code:

find /var/www -name "*.log" -type f -size +10M -exec echo > {} /;

any ideas?

stress_junkie · 11-14-2010, 08:29 PM

You should be able to configure logrotate to maintain the log files.

steve51184 · 11-14-2010, 08:41 PM

ahh... how do i do that?

stress_junkie · 11-14-2010, 08:43 PM

Install the logrotate software from your distribution's software package repository. It will probably already have a default configuration. It generally runs from the system's (root's) crontab files located in the /etc directory. Once it's installed the startup files explain how to customize it.

Most distributions come with logrotate already installed and configured by default. What distribution and version are you using?

steve51184 · 11-14-2010, 08:46 PM

Ubuntu server 8.10

stress_junkie · 11-14-2010, 08:48 PM

You should already have logrotate.

Code:

$ which logrotate

You should see

Code:

$ which logrotate
/usr/sbin/logrotate

Also

Code:

ls -ld /etc/log*
drwxr-xr-x 8 root root  4096 2010-11-04 10:31 /etc/logcheck
-rw-r--r-- 1 root root 10788 2010-01-26 12:09 /etc/login.defs
-rw-r--r-- 1 root root   599 2010-03-06 22:00 /etc/logrotate.conf
drwxr-xr-x 2 root root  4096 2010-11-14 08:33 /etc/logrotate.d
drwxr-xr-x 2 root root  4096 2010-11-04 10:31 /etc/logtool

Here the main configuration file is /etc/logrotate.conf but there are more files in /etc/logrotate.d to make fine tuning adjustments to logrotate's behavior.

Still. I'm surprised that it wasn't automatically running by default when you installed your OS.

steve51184 · 11-14-2010, 08:50 PM

yeah that's the output i have... how do i change the options?

stress_junkie · 11-14-2010, 08:56 PM

Check to see if logrotate is already in the system's crontab files.

Code:

$ find /etc/cron* -type f -exec grep -H logrotate {} \;
/etc/cron.daily/logrotate:test -x /usr/sbin/logrotate || exit 0
/etc/cron.daily/logrotate:/usr/sbin/logrotate /etc/logrotate.conf
/etc/cron.daily/exim4-base:        logrotate -f /etc/logrotate.d/exim4-paniclog
/etc/cron.monthly/standard:# rotation of wtmp and btmp taken over by logrotate

If it is then it should be running already. Then the question is why isn't it already rotating your http log files. You can see that these crontab files when logrotate will run. Look in /etc/logrotate.d and you will see that there is a separate file for each log file maintained by logrotate.

Code:

$ ls /etc/logrotate.d
apport      cups            jockey-common  rsyslog              wpa_supplicant
apt         dpkg            pm-utils       speech-dispatcher
aptitude    exim4-base      ppp            ufw
atop        exim4-paniclog  psaccs_atop    unattended-upgrades
consolekit  hobbit-client   psaccu_atop    wpa_action

Well, rsyslog has a lot of files listed in it. So each file in /etc/logrotate.d has a list of files for logrotate to maintain.

steve51184 · 11-14-2010, 09:07 PM

my crontab has this:

Quote:

/etc/cron.daily/logrotate:test -x /usr/sbin/logrotate || exit 0
/etc/cron.daily/logrotate:/usr/sbin/logrotate /etc/logrotate.conf
/etc/cron.monthly/standard:# rotation of wtmp and btmp taken over by logrotate

and after looking in /etc/logrotate.d/apache2 i see this line at the top:

Quote:

/var/log/apache2/*.log

and my logs are like this PER site:

/var/www/web1/log/error.log
/var/www/web1/log/web.log

and the web.log is a symlink to year/month/web.log that changes every month so this month it's /var/www/web1/log/2010/11/web.log

and i have 18 sites on this server so web1, web2, web3 etc

stress_junkie · 11-14-2010, 09:15 PM

I can see a light at the end of this tunnel.

Initially I would just add the full path to this year's web.logs. You can find out later if logrotate will follow a symlink. I imagine it will but I don't know. You could do two or three a day to make it more bearable and to make sure that logrotate doesn't have too much work to do the first time it processes each web site's logs.

I just noticed that you have your OS in your signature. Ooops.

steve51184 · 11-14-2010, 09:19 PM

Quote:

Originally Posted by stress_junkie

Initially I would just add the full path to this year's web.log. You can find out later if logrotate will follow a symlink. I imagine it will but I don't know.

well the problem is there monthly not yearly and there's 18 sites :/

Quote:

Originally Posted by stress_junkie

I just noticed that you have your OS in your signature. Ooops.

hehe lol

stress_junkie · 11-14-2010, 09:29 PM

Quote:

Originally Posted by steve51184

well the problem is there monthly not yearly

Then test it tonight to see if it will follow the symlink. That at least will get this month's log files rotated and it will automatically move to each new month's log files.

Quote:

Originally Posted by steve51184

and there's 18 sites :/

That wouldn't be a problem if you had done this from the beginning.

I don't think there's any way around the fact that you have some manual maintenance to do for the old log files. Maybe you could do something like this for the old ones

Code:

find /var/www/web* -type f -name *log -exec gzip {} \;

Maybe add a condition in the find command to only do files that are older than 30 days so you don't mess up the current log files.

Mind you logrotate will normally delete old log files so you could just delete old ones yourself instead of compressing them. Log file retention is configurable on a per log basis just in case you never want log files deleted. You don't have to have the old logs compressed either if you'd rather keep them in readable format. You could just have a new log started each day and leave the old log in its normal format and never have any logs deleted.

The more I think about it the more I think that's what you want to do. But logrotate won't split up the old log files. You'd probably want to use something like sed to do that.

steve51184 · 11-21-2010, 06:46 PM

hmm going back to my original idea is there a way i can just blank the log files and not use logrotate?

mrhhug · 01-12-2011, 02:40 AM

steve51184,

sounds to me like you want to simply remove these log files to free up hard drive space and your running ubuntu with apache; it should be quite easy. I am running maverick and all my error and access logs are in "/var/log/apache2/" . Delete away if you don't want the history of your webserver and just want disk space. You are reporting some serious log sizes, you can change the way apache makes logs if you are interested... If you are deleting these files through some GUI or anything remember to empty the trash.