LinuxQuestions.org - [SOLVED] kernel: init: rsyslog main process killed by ABRT signal -- how to investigate ?

- Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)

- - kernel: init: rsyslog main process killed by ABRT signal -- how to investigate ? (https://www.linuxquestions.org/questions/linux-newbie-8/kernel-init-rsyslog-main-process-killed-by-abrt-signal-how-to-investigate-4175419996/)

kernel: init: rsyslog main process killed by ABRT signal -- how to investigate ?

Hi,

I am fighting with rsyslog to find out why is it killed approximately each minute by ABRT signal. Could please anybody help me how to find what is causing it or/and why ?

messages:
2012-08-02T11:33:50.662054+01:00 localhost kernel: [12479.287923] init: rsyslog main process (25952) killed by ABRT signal
2012-08-02T11:33:50.662057+01:00 localhost kernel: [12479.287951] init: rsyslog main process ended, respawning

thank you for any information,
kind regards,
M.

Quote:

Originally Posted by masuch (Post 4744188)

I am fighting with rsyslog to find out why is it killed approximately each minute by ABRT signal. Could please anybody help me how to find what is causing it or/and why ?

Unfortunately not much to go on. FYI here's the Wikipedia description of SIGABRT and (asserting this is what you use) rsyslog Launchpad tickets (note Bug #1031962 "init: rsyslog main process killed by ABRT signal").

Questions:
- Did this happen due to an upgrade of Rsyslog or any of its dependencies? If so does reverting to the previous version work?
- Did this happen due to (re)configuration of /etc/rsyslog.conf? If so, what are the changes? And does a config check ("-f /path/to/configfile -N level") show everything is OK?
- Rsyslog allows for debug mode by starting it with the "-d" switch or sending SIGUSR1 to a running process. Does enabling debug mode show more nfo?
- Are there any other system changes that should be considered like software updates, AppArmor profiles or any other stuff?
* As a last resort, does starting Rsyslog from 'strace' ("-f -ff -o /path/to/logfile") show anything?

thanks a lot for information.

Quote:

note Bug #1031962 "init: rsyslog main process killed by ABRT signal"

bug was reported by me.

Quote:

- Did this happen due to an upgrade of Rsyslog or any of its dependencies? If so does reverting to the previous version work?

It did not happened by installation of rsyslog or any dependencies.

Quote:

- Did this happen due to (re)configuration of /etc/rsyslog.conf? If so, what are the changes? And does a config check ("-f /path/to/configfile -N level") show everything is OK?

errors:

Code:

rsyslogd -f /etc/rsyslog.conf -N 9

rsyslogd: version 5.8.6, config validation run (level 9), master config /etc/rsyslog.conf

rsyslogd: WARNING: rsyslogd is running in compatibility mode. Automatically generated config directives may interfer with your rsyslog.conf settings. We suggest upgrading your config and adding -c5 as the first rsyslogd option.

rsyslogd: the last error occured in /etc/rsyslog.d/relp.conf, line 2:"$InputRELPServerRun 20514"

rsyslogd: the last error occured in /etc/rsyslog.conf, line 71:"$IncludeConfig /etc/rsyslog.d/*.conf"

rsyslogd: CONFIG ERROR: could not interpret master config file '/etc/rsyslog.conf'. [try http://www.rsyslog.com/e/2124 ]

rsyslogd: Warning: backward compatibility layer added to following directive to rsyslog.conf: ModLoad immark

rsyslogd: Warning: backward compatibility layer added to following directive to rsyslog.conf: MarkMessagePeriod 1200

rsyslogd: Warning: backward compatibility layer added to following directive to rsyslog.conf: ModLoad imuxsock

but /etc/rsyslog.d/relp.conf is correct. do not know why config error happened ?
-c5 has been there.

Quote:

- Rsyslog allows for debug mode by starting it with the "-d" switch or sending SIGUSR1 to a running process. Does enabling debug mode show more nfo?

I have changed
RSYSLOGD_OPTIONS="-c5 -d" in /etc/default/rsyslog
but no any additional messages appeared so far.

Quote:

- Are there any other system changes that should be considered like software updates, AppArmor profiles or any other stuff?

I have added in AppArmor profile /etc/apparmor.d/usr.sbin.rsyslogd:
/var/rsyslog/** rw,

Quote:

* As a last resort, does starting Rsyslog from 'strace' ("-f -ff -o /path/to/logfile") show anything?

strace -f -ff -o rsyslog.strace > -ttt -T -s 100 rsyslogd -c5 -nd >rsyslog.debug
some errors:
1.
write(1, "open error 13, file '/var/rsyslog/work/dbq.00000001': Permission denied\n", 72) = 72 <0.000007>
-- I have created file manually with syslog:adm owner. seems working - some messages appeared there.
2.
write(1, "postgres query execution failed: PGRES_FATAL_ERROR\n", 51) = 51 <0.000759>
write(1, "7486.536509538:", 15) = 15 <0.000008>
write(1, "7f123cff9700: ", 14) = 14 <0.000007>
write(1, "pgsql, DBError(silent): db error (1): no connection to the server\n\n\n", 68) = 68 <0.000007>
...
stat("/home/syslog/.postgresql/postgresql.crt", 0x7f123cff5dd0) = -1 ENOENT (No such file or directory) <0.000009>
stat("/home/syslog/.postgresql/root.crt", 0x7f123cff5dd0) = -1 ENOENT (No such file or directory) <0.000009>
-- need to correct Postgre database connectivity and ... .
3.
write(1, "Called LogError, msg: db error (1146): Table 'Syslog.syslog_incoming' doesn't exist\n\n", 85) = 85 <0.000009>
-- has been corrected.
4.
write(1, "action 0x2078220 call returned -2007\n", 37) = 37 <0.000006>
write(1, "7439.534684181:", 15) = 15 <0.000006>
write(1, "7f1244fee700: ", 14) = 14 <0.000006>
write(1, "tryDoAction: unexpected error code -2007[nElem 1, Commited UpTo 0], finalizing\n", 79) = 79 <0.000006>
-- have no idea how to correct it ?
5.
rsyslogd: db error (1054): Unknown column 'invld' in 'field list'
-- have no idea how to correct it.
have found some info on http://kb.monitorware.com/100-cpu-ut...on-t10230.html
but apparently no solving at for long time.
I did not succeed to find the source causing this error.
6.
write(1, "cfline: '$ActionResumeRetryCount -1 # infinite retries on insert failure'\n", 75) = 75 <0.000005>
-- that should be correct because I have defined $ActionResumeRetryCount -1 so if no success that continue.
7.
open("/opt/oracle.instantclient_11_2.64b/libpq.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) <0.000006>
open("/opt/MonitorSoftware/lib/libpq.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) <0.000009>
open("/opt/AMDAPP/lib/x86_64/libpq.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) <0.000007>
open("/opt/AMDAPP/lib/x86/libpq.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) <0.000006>
-- created ln -s /usr/lib/libpq.so.5 libpq.so.5

killed by ABRT signal continues to appear :-(

I don't know what your /etc/rsyslog.d/*.conf and /etc/rsyslog.conf contain so I can't comment on or help you correct any of that. There's too much errors you can investigate and fix yourself before concluding Rsyslogd kills itself for some unknown reason.

Quote:

Originally Posted by unSpawn (Post 4746380)

I am sorry. I forgot to show content.

/etc/rsyslog.conf:

Quote:

$ModLoad imuxsock
$ModLoad imklog
$ModLoad imudp
$UDPServerRun 1025 # I changed it from 514 because of "permission problems for port under 1024"
$ModLoad imtcp
$InputTCPServerRun 1025
$RepeatedMsgReduction on
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog
$WorkDirectory /var/spool/rsyslog
$SystemLogRateLimitInterval 5
$SystemLogRateLimitBurst 2500
*.* /var/log/everything.log
$IncludeConfig /etc/rsyslog.d/*.conf

/etc/rsyslog.d/postfix.conf:

Quote:

$AddUnixListenSocket /var/spool/postfix/dev/log

/etc/rsyslog.d/mysql.conf

Code:

$ModLoad ommysql

*.* :ommysql:localhost,Syslog,rsyslog,password

$template cacti_syslog,"insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL

*.* >localhost,Syslog,rsyslog,password;cacti_syslog

$WorkDirectory /var/rsyslog/work

$ActionQueueType LinkedList 

$ActionQueueFileName dbq    

$ActionResumeRetryCount -1

/etc/rsyslog.d/pgsql.conf

Code:

$ModLoad ompgsql

*.* :ompgsql:127.0.0.1,Syslog,rsyslog,password

$template cacti_syslog,"insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL

*.* >localhost,Syslog,rsyslog,password;cacti_syslog

/etc/rsyslog.d/relp.conf

Quote:

$ModLoad imrelp
$InputRELPServerRun 20514

/etc/rsyslog.d/10-octopussy.conf

Quote:

$WorkDirectory /var/lib/octopussy/rsyslog
$CreateDirs on
$MaxMessageSize 8k
$ActionQueueMaxDiskSpace 1g
$ActionQueueFileName rsyslog
$ActionQueueHighWaterMark 250000
$ActionQueueLowWaterMark 200000
$ActionQueueType LinkedList # [FixedArray/LinkedList/Direct/Disk]
$ActionQueueSaveOnShutdown on
$ActionQueueWorkerThreads 1 # 1 cpu
*.* |/var/spool/octopussy/octo_fifo
:hostname, !isequal, "myhostname" ~

/etc/rsyslog.d/20-ufw.conf

Quote:

:msg,contains,"[UFW " /var/log/ufw.log
& ~

/etc/rsyslog.d/50-default.conf

Code:

auth,authpriv.*                        /var/log/auth.log

*.*;auth,authpriv.none        -/var/log/syslog

cron.*                                      /var/log/cron.log

daemon.*                                -/var/log/daemon.log

kern.*                                      -/var/log/kern.log

lpr.*                                        -/var/log/lpr.log

user.*                                      -/var/log/user.log

mail.*                -/var/log/mail.log

mail.info        -/var/log/mail.info

mail.warn        -/var/log/mail.warn

mail.err                /var/log/mail.err

news.crit                /var/log/news/news.crit

news.err                        /var/log/news/news.err

news.notice                -/var/log/news/news.notice

*.=debug;\

        auth,authpriv.none;\

        news.none;mail.none            -/var/log/debug

*.=info;*.=notice;*.=warn;\

        auth,authpriv.none;\

        cron,daemon.none;\

        mail,news.none                        -/var/log/messages

*.emerg                                :omusrmsg:*

daemon,mail.*;\

        news.=crit;news.=err;news.=notice;\

        *.=debug;*.=info;\

        *.=notice;*.=warn        /dev/tty8

daemon.*;mail.*;\

        news.err;\

        *.=debug;*.=info;\

        *.=notice;*.=warn        |/dev/xconsole

(At this moment I am fighting with rsyslogd: db error (1054): Unknown column 'invld' in 'field list'.
I understand that it is not safe to show to public my port mapping but I can change it any time and they should not be visible outside of my LAN network.
I do not need to push messages into mysql and postgre databases -just experimental - postgresql does not work yet.
As well I need to amend /dev/xconsole.
There should not be problem within latency of my SSD hard disk
(R/W IO operations are as fast as memory operations. I have this problem if I run the same OS instance cloned on SATA II/III hard disk/s.)

Please feel free to post any suggestions , recommendations , amendments, purging.
Thanks a lot for helping me.

Quote:

Originally Posted by masuch (Post 4746519)

I am sorry. I forgot to show content.

Configs look OK to me but I'm no Rsyslogd guru.

Quote:

Originally Posted by masuch (Post 4746519)

(At this moment I am fighting with rsyslogd: db error (1054): Unknown column 'invld' in 'field list'.

Last Adiscon thread I read the developers first suspicion was against MySQL insertion and later on in the same thread a suspicion was raised wrt (old school) Syslogd (not properly) defining or setting a wrong priority level that caused this.

Quote:

Originally Posted by masuch (Post 4746519)

I do not need to push messages into mysql and postgre databases -just experimental - postgresql does not work yet.

I suggest you move the MySQL and PostGreSQL configs out of your /etc/rsyslog.d/ directory and see if that helps.

Quote:

Originally Posted by masuch (Post 4746519)

As well I need to amend /dev/xconsole.

Please document changes and supply to distro or upstream?

Hi,

From that time I post something here
I compile new version 5.6.13 of rsyslog from git repository on recommendation to solve another problem - pthread_mutex_lock.c:62: __pthread_mutex_lock: Assertion `mutex->__data.__owner == 0' failed
(https://bugs.launchpad.net/ubuntu/+s...g/+bug/1024731)
- error did not appeared so far.

killed by ABRT signal did not appeared from that time either. I am not for 100 % sure what caused but seemed that problem was
due to missing chown syslog:adm /var/rsyslog/work (I have to more investigate)

/dev/xconsole. - I put
mknod -m 640 /dev/xconsole c 1 3
chown syslog:adm /dev/xconsole
in
/etc/init/rsyslog.conf
so problem with /dev/xconsole did not appeared again.

rsyslogd: db error (1054): Unknown column 'invld' in 'field list'.
did not solved yet.

I am still running in debug mode so I am now fighting with another problem which is logrotate - apparently does not work properly
within this configuration as it is confirmed on many other forums. - created my own logrotate running from crontab.

It is very difficult to read so huge files even for vim with file autoreloading functionality. Within 6 hours it has been generated 18Gbytes messages which was too much even for vim (I had to kill it by kill -9). sigterm did not work :-)

so ,basically this thread could be closed.

thanks a lot for helping me out.
M.