Central log server aggregating all messages to /var/log/messages
 

Good afternoon,

I have a central logging server and rsyslog is set up and functioning. However, all the remote /var/log/messages are being aggregated in the central log server's /var/log/messages. I have all the other logs going to the /var/log/rsyslog_custom/%HOSTNAME% directories.

I'm currently running Centos 6.5 and running rsyslog version 5.8. Here is a copy of my rsyslog.conf:

# rsyslog v5 configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog # provides kernel logging support (previously done by rklogd)
#$ModLoad immark # provides --MARK-- message capability

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514


#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf


#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
#*.info;mail.none;authpriv.none;cron.none /var/log/messages

# send everything to /var/log/messages
*.* /var/log/messages

# The authpriv file has restricted access.
#authpriv.* /var/log/secure

# Log all the mail messages in one place.
#mail.* -/var/log/maillog


# Log cron stuff
#cron.* /var/log/cron

# Everybody gets emergency messages
*.emerg *

# Save news errors of level crit and higher in a special file.
#uucp,news.crit /var/log/spooler

# Save boot messages also to boot.log
#local7.* /var/log/boot.log


# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###

# A template to for higher precision timestamps + severity logging
$template SpiceTmpl,"%TIMESTAMP%.%TIMESTAMP:::date-subseconds% %syslogtag% %syslogseverity-text%:%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"

:programname, startswith, "spice-vdagent" /var/log/spice-vdagent.log;SpiceTmpl

$template TmplAuth, "/var/log/rsyslog_custom/%HOSTNAME%/%PROGRAMNAME%"
$template TmplMsg, "/var/log/rsyslog_custom/%HOSTNAME%/%PROGRAMNAME%"
authpriv.* ?TmplAuth
*.info,mail.none,authpriv.none,cron.none ?TmplMsg

Like I stated above, all the other log files are going where they need to be going. It is only the messages log that is aggregating into the local /var/log/messages.

In my research I have seen a couple of possibilities to handle this but none have flushed out. One that looked very promising was adding an if statement before all the rules (http://www.rsyslog.com/storing-messa...specific-file/), but that didn't work. But I think it was going down the right road. I used the following:

if $HOSTNAME !=

if $fromhost-ip != '123.4.56.78' then '/var/log/rsyslog_custom/%HOSTNAME%/%PROGRAMNAME%'
& ~

This stops the logging to the local /var/log/messages but no messages file is created. So, I thought i would play a little more. And tried this:

if $fromhost-ip != '123.4.56.78' then '/var/log/rsyslog_custom/%HOSTNAME%/messages'
& ~

Any help would be greatly appreciated. Thank you all!

Sincerely,
Lee

...and that isn't odd considering you have traditional local stanzas like precede "remote" rules. Move those to the bottom of the conf and see if that works for you. If it does please mark this thread "solved".

unSpawn,

Thank you for your reply. So, if I understand you correctly, if I move the remote rules ahead of the local rules that should fix the problem, correct? Or just the *.* /var/log/messages to below the remote rules?

Thanks!
Lee

First option, yes.

Still logging to /var/log/messages
 

unSpawn,

Thanks for the clarification. I moved the templates to above the default rule set and still have the same problem.

Any other recommendations?

Sincerely,
Lee

All,

I've figured it out. Thanks to unSpawn for starting me in the correct direction. Here is my latest version of rsyslog.conf (changes are in bold):

# rsyslog v5 configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog # provides kernel logging support (previously done by rklogd)
#$ModLoad immark # provides --MARK-- message capability

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514

#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

#################################################################################################### ###########
# This next section will place all remote logging files in their own subdirectory to /var/log/HOSTS. #
#################################################################################################### ###########

$template TmplAuth, "/var/log/HOSTS/%HOSTNAME%/%PROGRAMNAME%"
$template TmplMsg, "/var/log/HOSTS/%HOSTNAME%/%PROGRAMNAME%"
$template Msgs, "/var/log/HOSTS/%HOSTNAME%/messages"
authpriv.* ?TmplAuth
*.info,mail.none,authpriv.none,cron.none ?TmplMsg
*.* ?Msgs
& ~

#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages

# send everything to /var/log/messages
*.* /var/log/messages

# The authpriv file has restricted access.
#authpriv.* /var/log/secure

# Log all the mail messages in one place.
#mail.* -/var/log/maillog

# Log cron stuff
#cron.* /var/log/cron

# Everybody gets emergency messages
*.emerg *

# Save news errors of level crit and higher in a special file.
#uucp,news.crit /var/log/spooler

# Save boot messages also to boot.log
#local7.* /var/log/boot.log

# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###

# A template to for higher precision timestamps + severity logging
$template SpiceTmpl,"%TIMESTAMP%.%TIMESTAMP:::date-subseconds% %syslogtag% %syslogseverity-text%:%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"

:programname, startswith, "spice-vdagent" /var/log/spice-vdagent.log;SpiceTmpl

Sincerely,
Lee

lhiggie1: Thanks for posting your update and fix, this helped me out. And just to elaborate, placement is EVERYTHING, as unSpawn stated earlier in the thread.