[SOLVED] rsyslog logs to syslog.1 instead of syslog

tuxthegreat · 05-06-2018, 01:26 PM

Distro - Ubuntu Mate 16.04.4 / Kernel 4.13.0-39-generic

rsyslog version 8.34.0-oadiscon2xenial1 - Not the 8.16.0 ver that is shipped with the standard distro.

The reason I am using a ppa rsyslog https://launchpad.net/~adiscon/+arch...untu/v8-stable, rather than the current 8.16.0 one that comes with 16.04.4 is that the 8.16 one caused a daily kernel panic at the same time, when the logrotation would do it's daily thing.
The last piece of syslog would look a lot like this guy from RedHat is having

Quote:

Jul 30 20:19:11 hostname rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="1457" x-info="http://www.rsyslog.com"] rsyslogd was HUPed

This was repeated several times, maybe 50-100 before total system non-responsiveness. I have other reasons for using this version, mainly for the plethora of add-ons that come with this rsyslog repo, such as logcat module, the mmutf8fix module which deals with non-UTF character sets ie: ISO 8859. And another 20~ different modules. Here are my logrotate configs -> cat /etc/rsyslog.conf

Code:

#  /etc/rsyslog.conf	Configuration file for rsyslog.
#
#			For more information see
#			/usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
#
#  Default logging rules can be found in /etc/rsyslog.d/50-default.conf


#################
#### MODULES ####
#################

$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
#$ModLoad immark  # provides --MARK-- message capability

# provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514


###########################
#### GLOBAL DIRECTIVES ####
###########################

#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Filter duplicated messages
$RepeatedMsgReduction on

#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog

#
# Where to place spool files
#
$WorkDirectory /var/spool/rsyslog

#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf

-> cat /etc/logrotate.d/rsyslog

Code:

/var/log/syslog
{
	rotate 7
	daily
	missingok
	notifempty
	delaycompress
	compress
	postrotate
		invoke-rc.d rsyslog rotate >/dev/null
	endscript
}

/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/rsyslog.log
/var/log/debug
/var/log/messages
{
	rotate 4
	weekly
	missingok
	notifempty
	compress
	delaycompress
	sharedscripts
	postrotate
		invoke-rc.d rsyslog rotate >/dev/null
	endscript
}

-> cat /etc/init/rsyslog.conf

Code:

# rsyslog - system logging daemon
#
# rsyslog is an enhanced multi-threaded replacement for the traditional
# syslog daemon, logging messages from applications

description	"system logging daemon"

start on filesystem
stop on runlevel [06]

expect fork
kill timeout 30
respawn

pre-start script
    /lib/init/apparmor-profile-load usr.sbin.rsyslogd
end script

script
    . /etc/default/rsyslog
    exec rsyslogd $RSYSLOGD_OPTIONS
end script

-> cat /etc/rsyslog.d/50-default.conf

Code:

#  Default rules for rsyslog.
#
#			For more information see rsyslog.conf(5) and /etc/rsyslog.conf

#
# First some standard log files.  Log by facility.
#
auth,authpriv.*			/var/log/auth.log
*.*;auth,authpriv.none		/var/log/syslog
syslog.*                        /var/log/rsyslog.log #rsyslog error messages
#cron.*				/var/log/cron.log
#daemon.*			/var/log/daemon.log
kern.*				/var/log/kern.log
#lpr.*				/var/log/lpr.log
mail.*				/var/log/mail.log
#user.*				/var/log/user.log

#
# Logging for the mail system.  Split it up so that
# it is easy to write scripts to parse these files.
#
#mail.info			/var/log/mail.info
#mail.warn			/var/log/mail.warn
mail.err			/var/log/mail.err

#
# Logging for INN news system.
#
news.crit			/var/log/news/news.crit
news.err			/var/log/news/news.err
news.notice			/var/log/news/news.notice

#
# Some "catch-all" log files.
#
#*.=debug;\
#	auth,authpriv.none;\
#	news.none;mail.none	/var/log/debug
#*.=info;*.=notice;*.=warn;\
#	auth,authpriv.none;\
#	cron,daemon.none;\
#	mail,news.none		/var/log/messages

#
# Emergencies are sent to everybody logged in.
#
*.emerg                                :omusrmsg:*

#
# I like to have messages displayed on the console, but only on a virtual
# console I usually leave idle.
#
#daemon,mail.*;\
#	news.=crit;news.=err;news.=notice;\
#	*.=debug;*.=info;\
#	*.=notice;*.=warn	/dev/tty8

# The named pipe /dev/xconsole is for the `xconsole' utility.  To use it,
# you must invoke `xconsole' with the `-file' option:
#
#    $ xconsole -file /dev/xconsole [...]
#
# NOTE: adjust the list below, or you'll go crazy if you have a reasonably
#      busy site..
#
#
# As this functionality is almost never needed, it is commented out. If you
# need it, be sure to remove the comment characters below.
#daemon.*;mail.*;\
#	news.err;\
#	*.=debug;*.=info;\
#	*.=notice;*.=warn	|/dev/xconsole

If I am missing any conf files that you guys want to see, please let me know, otherwise what if anything can be done? rsyslog *works* as in it logs, and doesn't cause a kernel panic anymore, but it logs it to the wrong place

Any help is much appreciated.

RandomTroll · 05-06-2018, 04:30 PM

logrotate archives old logs to .1 (syslog.1, messages.1, debug.1 ...), shifting the numbers up (syslog.1 => syslog.2) until it deletes aged-out logs. Does that apply to your situation?

tuxthegreat · 05-06-2018, 05:00 PM

Quote:

Originally Posted by RandomTroll

logrotate archives old logs to .1 (syslog.1, messages.1, debug.1 ...), shifting the numbers up (syslog.1 => syslog.2) until it deletes aged-out logs. Does that apply to your situation?

No, it's like this syslog, syslog.1, syslog.2.gz, syslog.3.gz and up until 7.gz. The syslog remains empty after each daily rotation. I can always restart rsyslog by service rsyslog restart and it starts to log to syslog again, but then it works ok until the next rotation, which is daily.

RandomTroll · 05-06-2018, 08:22 PM

So logrotate archives daily, deletes archives more than a week old. What you report sounds like rsyslog isn't stopping (or restarting) on rotation. I use syslog, not rsyslog, so your experience may be different, but if I try to archive a syslog target file without stopping syslogd first it doesn't start a new set of log files but fails to record any new entries. I don't know why your logrotate wouldn't get it right. I started archiving log files before logrotate existed so learned this through practice. Try running a

Code:

kill -sighup

on rsyslog after logrotate, at least as a diagnostic.

tuxthegreat · 05-06-2018, 08:33 PM

Quote:

Originally Posted by RandomTroll

So logrotate archives daily, deletes archives more than a week old. What you report sounds like rsyslog isn't stopping (or restarting) on rotation. I use syslog, not rsyslog, so your experience may be different, but if I try to archive a syslog target file without stopping syslogd first it doesn't start a new set of log files but fails to record any new entries. I don't know why your logrotate wouldn't get it right. I started archiving log files before logrotate existed so learned this through practice. Try running a

Code:

kill -sighup

on rsyslog after logrotate, at least as a diagnostic.

Ok I will report my findings tomorrow, well *today* later on after I sleep and when it's time the rotation happens.

tuxthegreat · 05-06-2018, 10:24 PM

Ok it seems that logrotate isn't the issue here, I checked my syslog and it's empty again, everything went to syslog.1 . There are some errors in my syslog.1, perhaps there is a clue there

Code:

May  7 02:16:34 ### systemd[1]: Starting Stop ureadahead data collection...
May  7 02:16:34 ### systemd[1]: Stopping Read required files in advance...
May  7 02:16:34 ### systemd[1]: Started Stop ureadahead data collection.
May  7 02:16:34 ### ureadahead[261]: ureadahead:events/fs/open_exec/enable: Ignored relative path
May  7 02:16:34 ### ureadahead[261]: ureadahead:buffer_size_kb: Ignored relative path
May  7 02:16:34 ### ureadahead[261]: ureadahead:tracing_on: Ignored relative path
May  7 02:16:34 ### ureadahead[261]: ureadahead:.: Ignored relative path
May  7 02:16:34 ### ureadahead[261]: message repeated 18 times: [ ureadahead:.: Ignored relative path]
May  7 02:16:34 ### ureadahead[261]: ureadahead:size: Ignored relative path
May  7 02:16:34 ### ureadahead[261]: ureadahead:start: Ignored relative path
May  7 02:16:34 ### ureadahead[261]: ureadahead:size: Ignored relative path
May  7 02:16:34 ### ureadahead[261]: ureadahead:start: Ignored relative path
May  7 02:16:34 ### ureadahead[261]: ureadahead:size: Ignored relative path
May  7 02:16:34 ### ureadahead[261]: ureadahead:start: Ignored relative path
May  7 02:16:34 ### ureadahead[261]: ureadahead:size: Ignored relative path
May  7 02:16:34 ### ureadahead[261]: ureadahead:start: Ignored relative path
May  7 02:16:34 ### ureadahead[261]: ureadahead:size: Ignored relative path
May  7 02:16:34 ### ureadahead[261]: ureadahead:start: Ignored relative path
May  7 02:16:34 ### ureadahead[261]: ureadahead:.: Ignored relative path
May  7 02:16:34 ### ureadahead[261]: ureadahead:: Ignored relative path
May  7 02:16:34 ### ureadahead[261]: ureadahead:1/stat: Ignored relative path
May  7 02:16:34 ### ureadahead[261]: ureadahead:1/cmdline: Ignored relative path
May  7 02:16:34 ### ureadahead[261]: ureadahead:2/stat: Ignored relative path
May  7 02:16:34 ### ureadahead[261]: ureadahead:2/cmdline: Ignored relative path
May  7 02:16:34 ### ureadahead[261]: ureadahead:3/stat: Ignored relative path
May  7 02:16:34 ### ureadahead[261]: ureadahead:3/cmdline: Ignored relative path
May  7 02:16:34 ### ureadahead[261]: ureadahead:4/stat: Ignored relative path

This goes on for quite a while, about 2 thousand lines and then I get these

Code:

May  7 02:16:34 ### ureadahead[261]: ureadahead:start: Ignored relative path
May  7 02:16:34 ### ureadahead[261]: ureadahead:size: Ignored relative path

Maybe 200 of those. Reason I mentioned these errors, is the time I think the syslog went bonkers and switched to syslog.1.

RandomTroll · 05-06-2018, 11:30 PM

Quote:

Originally Posted by tuxthegreat

I think the syslog went bonkers and switched to syslog.1.

As though it reached a maximum size then rotated itself?

I separate loggable events by daemon. For instance I re-write dhcpcd and acpi to use their own logs. I try to leave syslog & message to unusual events. This is easier with rsyslog because rsyslog.conf allows you to divert entries by daemon. Separate those systemd and ureadahead entries.

tuxthegreat · 05-07-2018, 10:42 AM

Quote:

Originally Posted by RandomTroll

As though it reached a maximum size then rotated itself?

I separate loggable events by daemon. For instance I re-write dhcpcd and acpi to use their own logs. I try to leave syslog & message to unusual events. This is easier with rsyslog because rsyslog.conf allows you to divert entries by daemon. Separate those systemd and ureadahead entries.

Ok good point, I added a little addition ( size 500M ) to the rsyslog config as such, let's see if this works.

Code:

/var/log/syslog
{
        rotate 7
        size 500M
        daily
        missingok
        notifempty
        delaycompress
        compress
        postrotate
                invoke-rc.d rsyslog rotate >/dev/null
        endscript
}

tuxthegreat · 05-07-2018, 03:37 PM

Quote:

Originally Posted by RandomTroll

As though it reached a maximum size then rotated itself?

I separate loggable events by daemon. For instance I re-write dhcpcd and acpi to use their own logs. I try to leave syslog & message to unusual events. This is easier with rsyslog because rsyslog.conf allows you to divert entries by daemon. Separate those systemd and ureadahead entries.

Looks like you hit the nail on the head with this one

I checked my syslog just now by using du -h syslog and it's 1.4M, it never grew above 1M before I noticed, so I guess that's one problem solved. Thank you for your patience, I know I can be a pain in the ..with my endless barrage of questions, now onto the next problem.