Log rotation..

Skillz · 08-30-2011, 02:51 AM

So it seems that my server isn't rotating logs like it should. I can confirm this as the logs ending with .[x] where [x] is a number usually 1 - 4 are all months old according to the time stamp and the main log file without any .[x] appended to it is rather large. I only noticed this because I have my messages log emailed to me daily using a script

Code:

less /var/log/messages | grep "$(date --date="yesterday" '+%b %e')" | mail -s "title of email" my@email.address

However, since I installed this script a year ago, I'm starting to see logs from last August show up in today's emails; so I began looking.

I know logrotate is what handles log rotation, but how can I check to see why it's stopped working for whatever reason?

I ran this command, as I figured it would shed some light on it but it doesn't seem to be helping.

/usr/sbin/logrotate -vf /etc/logrotate.conf

This is the output of the above command.

Code:

reading config file /etc/logrotate.conf
including /etc/logrotate.d
reading config file acpid
reading config info for /var/log/acpid
reading config file conman
reading config info for /var/log/conman/*
olddir is now /var/log/conman.old/
reading config file cups
reading config info for /var/log/cups/*_log
reading config file exim
reading config info for /var/log/exim_mainlog
reading config info for /var/log/exim_paniclog
reading config info for /var/log/exim_rejectlog
reading config file httpd
reading config info for /var/log/httpd/*log
reading config file mgetty
reading config info for /var/log/mgetty.log.tty[^.] /var/log/mgetty.log.tty[^.][^.] /var/log/mgetty.log.tty[^.][^.][^.] /var/log/mgetty.log.tty[^.][^.][^.][^.] /var/log/mgetty.log.tty[^.][^.][^.][^.][^.] /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.] /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.][^.] /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.][^.][^.] /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.][^.][^.][^.] /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.][^.][^.][^.][^.] /var/log/mgetty.log.unknown /var/log/mgetty.callback
reading config file mysql
reading config info for /var/lib/mysql/mysqld.log
reading config file named
reading config info for /var/log/named.log
reading config file ppp
reading config info for /var/log/ppp/connect-errors
reading config file psacct
reading config info for /var/account/pacct
reading config file rpm
reading config info for /var/log/rpmpkgs
reading config file sa-update
reading config info for /var/log/sa-update.log
reading config file samba
reading config info for /var/log/samba/*.log
reading config file setroubleshoot
reading config info for /var/log/setroubleshoot/*.log
reading config file snort
error: error accessing /var/log/snort/*: No such file or directory
error: snort:4 glob failed for /var/log/snort/*/*log

Not sure what most of that means, but whatever. Where do I start?

centosboy · 08-30-2011, 03:13 AM

Quote:

Originally Posted by Skillz

So it seems that my server isn't rotating logs like it should. I can confirm this as the logs ending with .[x] where [x] is a number usually 1 - 4 are all months old according to the time stamp and the main log file without any .[x] appended to it is rather large. I only noticed this because I have my messages log emailed to me daily using a script

Code:

less /var/log/messages | grep "$(date --date="yesterday" '+%b %e')" | mail -s "title of email" my@email.address

However, since I installed this script a year ago, I'm starting to see logs from last August show up in today's emails; so I began looking.

I know logrotate is what handles log rotation, but how can I check to see why it's stopped working for whatever reason?

I ran this command, as I figured it would shed some light on it but it doesn't seem to be helping.

/usr/sbin/logrotate -vf /etc/logrotate.conf

This is the output of the above command.

Code:

reading config file /etc/logrotate.conf
including /etc/logrotate.d
reading config file acpid
reading config info for /var/log/acpid
reading config file conman
reading config info for /var/log/conman/*
olddir is now /var/log/conman.old/
reading config file cups
reading config info for /var/log/cups/*_log
reading config file exim
reading config info for /var/log/exim_mainlog
reading config info for /var/log/exim_paniclog
reading config info for /var/log/exim_rejectlog
reading config file httpd
reading config info for /var/log/httpd/*log
reading config file mgetty
reading config info for /var/log/mgetty.log.tty[^.] /var/log/mgetty.log.tty[^.][^.] /var/log/mgetty.log.tty[^.][^.][^.] /var/log/mgetty.log.tty[^.][^.][^.][^.] /var/log/mgetty.log.tty[^.][^.][^.][^.][^.] /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.] /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.][^.] /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.][^.][^.] /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.][^.][^.][^.] /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.][^.][^.][^.][^.] /var/log/mgetty.log.unknown /var/log/mgetty.callback
reading config file mysql
reading config info for /var/lib/mysql/mysqld.log
reading config file named
reading config info for /var/log/named.log
reading config file ppp
reading config info for /var/log/ppp/connect-errors
reading config file psacct
reading config info for /var/account/pacct
reading config file rpm
reading config info for /var/log/rpmpkgs
reading config file sa-update
reading config info for /var/log/sa-update.log
reading config file samba
reading config info for /var/log/samba/*.log
reading config file setroubleshoot
reading config info for /var/log/setroubleshoot/*.log
reading config file snort
error: error accessing /var/log/snort/*: No such file or directory
error: snort:4 glob failed for /var/log/snort/*/*log

Not sure what most of that means, but whatever. Where do I start?

Best place to start is by running in debug mode

Code:

logrotate -v -d /path/to/logrotate.conf

from what you have pasted, looks like a config problem with the snort log directory, and so the whole process is failing.
Run again in debug mode and paste the output...or fix the error on line 4 of the snort config file.

Code:

error: error accessing /var/log/snort/*: No such file or directory
error: snort:4 glob failed for /var/log/snort/*/*log

[/code]

jlinkels · 08-30-2011, 03:31 AM

IIRC if in the logrotate process one of the logrotates fail (that is a file to rotate does not exist) all following commands are not executed. You would have this when you stopp logging with a certain program and delete old log files. That is a bit unexpected, as one would not consider that an error. Still logrotate does and ceases execution.

jlinkels

Skillz · 08-30-2011, 11:19 PM

Well I just deleted the snort config file for right now.

When I run it in debug mode, too much information is passing through the screen at once.

Tried running

logrotate -v -d /path/to/logrotate.conf | less

Still wouldn't stop so I could see everything.

Also tried

logrotate -v -d /path/to/logrotate.conf > output.txt

But it was just a blank document it created.

Skillz · 08-30-2011, 11:25 PM

Well I think log rotation has resumed. I will keep an eye on it for the next few days, then see about getting snort back in the mix. For the record, here is the snort config file.

Code:

# /etc/logrotate.d/snort
# $Id$

/var/log/snort/alert /var/log/snort/*log /var/log/snort/*/alert /var/log/snort/*/*log  {
    daily
    rotate 7
    missingok
    compress
    sharedscripts
    postrotate
        /etc/init.d/snortd restart 1>/dev/null || true
    endscript
}

Skillz · 08-30-2011, 11:29 PM

Actually, think I see the problem.

/var/log/snort/alert (E)xists

/var/log/snort/*log (No files end in log)

/var/log/snort/*/alert (their are no directories in /snort)

/var/log/snort/*/*log (their are no directories in /snort)

So I think if I change it to

/var/log/snort/*log* (Some files do have log as an extension, but behind it is a series of numbers [guessing its a time stamp])
/var/log/snort/alert* (This should gather all the files that are alert.{series of numbers})
delete this bit of code, as it wont catch anything that the /var/log/snort/*log* doesn't already catch.

Maybe?

jlinkels · 08-31-2011, 01:05 PM

I think you are creating problems with using *log* as files will get rotated and re-rotated. Check the man page (there is a paragraph on this) and refer to the olddir directive.

You redirection into a file to trace the error created a blank file because these error messages appear on stderr, not stdout.

Try: logrotate -v -d /path/to/logrotate.conf >> output.txt 2>&1

jlinkels

Skillz · 09-03-2011, 04:49 PM

Everything is working like it should, thanks everyone.