LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 05-08-2013, 09:20 PM   #1
hattori.hanzo
Member
 
Registered: Aug 2006
Posts: 168

Rep: Reputation: 15
rsyslog writes to disk & log rotation


Hello,

I am using rsyslog to received syslog messages from my firewalls. I have setup logrotation to happen as per rsyslog's example:

Code:
/etc/rsyslog.conf:

#####################################################
# Log everything to a per host daily logfile        #
#####################################################
$template DailyPerHostLogs,"/var/log/rsyslog/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%.log"
*.* -?DailyPerHostLogs

/etc/cron.hourly/syslog-bzip2:

# Compress *.log-files not changed in more than 24 hours:
find /var/log/syslog/2008 -type f -mtime +1  -name "*.log" -exec gzip '{}' \;
The cron job checks if the log file has not changed in more than 24 hours to them gzip it. Since the firewalls are very active and always logging packets this condition will never be true.

How can I do log rotation but not drop any syslog events? I did a test by removing -mtime +1 and it dropped about 7 seconds of logs.

I looked at rsyslog's disk queuing but that only applies if the destination is unavaliable but that will never be when it writes to disk.

Thanks.
 
Old 05-08-2013, 11:17 PM   #2
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.9, Centos 7.3
Posts: 17,362

Rep: Reputation: 2377Reputation: 2377Reputation: 2377Reputation: 2377Reputation: 2377Reputation: 2377Reputation: 2377Reputation: 2377Reputation: 2377Reputation: 2377Reputation: 2377
Try -mmin http://linux.die.net/man/1/find
 
Old 05-08-2013, 11:38 PM   #3
hattori.hanzo
Member
 
Registered: Aug 2006
Posts: 168

Original Poster
Rep: Reputation: 15
Thanks Chris. Will try that.
 
Old 05-09-2013, 12:16 AM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,353
Blog Entries: 55

Rep: Reputation: 3541Reputation: 3541Reputation: 3541Reputation: 3541Reputation: 3541Reputation: 3541Reputation: 3541Reputation: 3541Reputation: 3541Reputation: 3541Reputation: 3541
Quote:
Originally Posted by hattori.hanzo View Post
How can I do log rotation but not drop any syslog events?
You already let Rsyslogd handle log rotation by using a per-day, per-host log file name template.
Ergo Rsyslogd will close the old file and open a new one on day start.
So the Rsyslogd part is not what you're having problems with.


Quote:
Originally Posted by hattori.hanzo View Post
I looked at rsyslog's disk queuing but that only applies if the destination is unavaliable but that will never be when it writes to disk.
About $MainMsgQueue.*: "In this mode, receiver and output modules are de-coupled via an in-memory queue. This queue buffers messages when the output modules are not capable to process them as fast as they are received". So it's not availability of the output module but speed that dictates it.


Depending on how many hosts you need to handle, how long log files need to be stored and where they should be stored after processing I would rewrite the template as it IMHO makes no sense duplicating %HOSTNAME and having that much tree depth. With your current directory structure I would rewrite the cronjob this way:
Code:
#!/bin/sh --
# /var/log/rsyslog/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%.log"
DATE_STRING=$(/bin/date +"%Y/%m/%d" --date="yesterday")

find /var/log/syslog/ -type d -iname \*${DATE_STRING}\* | while read LOG_LOG_DIRNAME; do
 # Loop over yesterdays logs:
 find "${LOG_LOG_DIRNAME}" -type f | while read OLD_LOG_FILENAME; do
  # Superfluous now but loop only over closed files anyway:
  /sbin/fuser "${OLD_LOG_FILENAME}" >/dev/null 2>&1
  # Check if you want reporting else uncomment and add " || bzip2 "${OLD_LOG_FILENAME}"" to the line above
  case $? in
   0) /usr/bin/logger -t logrotate.custom "In use: "${OLD_LOG_FILENAME}".";;
   1) bzip2 "${OLD_LOG_FILENAME}";;
   *) /usr/bin/logger -t logrotate.custom "Unknown problem: "${OLD_LOG_FILENAME}". Investigate.";;
  esac
 done
done
exit 0
Yes it's convoluted but looping only over yesterdays logs and checking if they're not in use IMHO is the safest way to rotate these logs.
 
1 members found this post helpful.
Old 05-17-2013, 01:33 AM   #5
hattori.hanzo
Member
 
Registered: Aug 2006
Posts: 168

Original Poster
Rep: Reputation: 15
Thanks alot for the input. I will look into this further with some more testing.

Cheers.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Log Rotation with Date & Timstamp for Rotated File Linux_CTL_2013 Linux - General 3 05-02-2013 12:44 AM
rsyslog: what time does rotation happen? elfoozo Debian 6 08-24-2009 02:04 PM
How to change Debian log rotation of syslog and daemon.log onmountain Linux - Newbie 2 07-31-2008 02:27 AM
how to write log data to disk when the server is down while using rsyslog prakash.akumalla Linux - Newbie 1 06-25-2008 01:15 AM
Log Rotation for snort log does not seem to be working CentOS4.5 JasonKretzer Linux - Security 3 06-25-2007 12:25 PM


All times are GMT -5. The time now is 12:19 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration