kernel: init: rsyslog main process killed by ABRT signal -- how to investigate ?
I am fighting with rsyslog to find out why is it killed approximately each minute by ABRT signal. Could please anybody help me how to find what is causing it or/and why ?
2012-08-02T11:33:50.662054+01:00 localhost kernel: [12479.287923] init: rsyslog main process (25952) killed by ABRT signal
2012-08-02T11:33:50.662057+01:00 localhost kernel: [12479.287951] init: rsyslog main process ended, respawning
thank you for any information,
- Did this happen due to an upgrade of Rsyslog or any of its dependencies? If so does reverting to the previous version work?
- Did this happen due to (re)configuration of /etc/rsyslog.conf? If so, what are the changes? And does a config check ("-f /path/to/configfile -N level") show everything is OK?
- Rsyslog allows for debug mode by starting it with the "-d" switch or sending SIGUSR1 to a running process. Does enabling debug mode show more nfo?
- Are there any other system changes that should be considered like software updates, AppArmor profiles or any other stuff?
* As a last resort, does starting Rsyslog from 'strace' ("-f -ff -o /path/to/logfile") show anything?
thanks a lot for information.
-c5 has been there.
RSYSLOGD_OPTIONS="-c5 -d" in /etc/default/rsyslog
but no any additional messages appeared so far.
write(1, "open error 13, file '/var/rsyslog/work/dbq.00000001': Permission denied\n", 72) = 72 <0.000007>
-- I have created file manually with syslog:adm owner. seems working - some messages appeared there.
write(1, "postgres query execution failed: PGRES_FATAL_ERROR\n", 51) = 51 <0.000759>
write(1, "7486.536509538:", 15) = 15 <0.000008>
write(1, "7f123cff9700: ", 14) = 14 <0.000007>
write(1, "pgsql, DBError(silent): db error (1): no connection to the server\n\n\n", 68) = 68 <0.000007>
stat("/home/syslog/.postgresql/postgresql.crt", 0x7f123cff5dd0) = -1 ENOENT (No such file or directory) <0.000009>
stat("/home/syslog/.postgresql/root.crt", 0x7f123cff5dd0) = -1 ENOENT (No such file or directory) <0.000009>
-- need to correct Postgre database connectivity and ... .
write(1, "Called LogError, msg: db error (1146): Table 'Syslog.syslog_incoming' doesn't exist\n\n", 85) = 85 <0.000009>
-- has been corrected.
write(1, "action 0x2078220 call returned -2007\n", 37) = 37 <0.000006>
write(1, "7439.534684181:", 15) = 15 <0.000006>
write(1, "7f1244fee700: ", 14) = 14 <0.000006>
write(1, "tryDoAction: unexpected error code -2007[nElem 1, Commited UpTo 0], finalizing\n", 79) = 79 <0.000006>
-- have no idea how to correct it ?
rsyslogd: db error (1054): Unknown column 'invld' in 'field list'
-- have no idea how to correct it.
have found some info on http://kb.monitorware.com/100-cpu-ut...on-t10230.html
but apparently no solving at for long time.
I did not succeed to find the source causing this error.
write(1, "cfline: '$ActionResumeRetryCount -1 # infinite retries on insert failure'\n", 75) = 75 <0.000005>
-- that should be correct because I have defined $ActionResumeRetryCount -1 so if no success that continue.
open("/opt/oracle.instantclient_11_2.64b/libpq.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) <0.000006>
open("/opt/MonitorSoftware/lib/libpq.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) <0.000009>
open("/opt/AMDAPP/lib/x86_64/libpq.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) <0.000007>
open("/opt/AMDAPP/lib/x86/libpq.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) <0.000006>
-- created ln -s /usr/lib/libpq.so.5 libpq.so.5
killed by ABRT signal continues to appear :-(
I don't know what your /etc/rsyslog.d/*.conf and /etc/rsyslog.conf contain so I can't comment on or help you correct any of that. There's too much errors you can investigate and fix yourself before concluding Rsyslogd kills itself for some unknown reason.
(At this moment I am fighting with rsyslogd: db error (1054): Unknown column 'invld' in 'field list'.
I understand that it is not safe to show to public my port mapping but I can change it any time and they should not be visible outside of my LAN network.
I do not need to push messages into mysql and postgre databases -just experimental - postgresql does not work yet.
As well I need to amend /dev/xconsole.
There should not be problem within latency of my SSD hard disk
(R/W IO operations are as fast as memory operations. I have this problem if I run the same OS instance cloned on SATA II/III hard disk/s.)
Please feel free to post any suggestions , recommendations , amendments, purging.
Thanks a lot for helping me.
From that time I post something here
I compile new version 5.6.13 of rsyslog from git repository on recommendation to solve another problem - pthread_mutex_lock.c:62: __pthread_mutex_lock: Assertion `mutex->__data.__owner == 0' failed
- error did not appeared so far.
killed by ABRT signal did not appeared from that time either. I am not for 100 % sure what caused but seemed that problem was
due to missing chown syslog:adm /var/rsyslog/work (I have to more investigate)
/dev/xconsole. - I put
mknod -m 640 /dev/xconsole c 1 3
chown syslog:adm /dev/xconsole
so problem with /dev/xconsole did not appeared again.
rsyslogd: db error (1054): Unknown column 'invld' in 'field list'.
did not solved yet.
I am still running in debug mode so I am now fighting with another problem which is logrotate - apparently does not work properly
within this configuration as it is confirmed on many other forums. - created my own logrotate running from crontab.
It is very difficult to read so huge files even for vim with file autoreloading functionality. Within 6 hours it has been generated 18Gbytes messages which was too much even for vim (I had to kill it by kill -9). sigterm did not work :-)
so ,basically this thread could be closed.
thanks a lot for helping me out.
|All times are GMT -5. The time now is 01:34 AM.|