LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 08-02-2012, 11:10 AM   #1
masuch
Member
 
Registered: Sep 2011
Location: /dev/null
Distribution: ubuntu 64bits
Posts: 135

Rep: Reputation: 1
Question kernel: init: rsyslog main process killed by ABRT signal -- how to investigate ?


Hi,

I am fighting with rsyslog to find out why is it killed approximately each minute by ABRT signal. Could please anybody help me how to find what is causing it or/and why ?

messages:
2012-08-02T11:33:50.662054+01:00 localhost kernel: [12479.287923] init: rsyslog main process (25952) killed by ABRT signal
2012-08-02T11:33:50.662057+01:00 localhost kernel: [12479.287951] init: rsyslog main process ended, respawning


thank you for any information,
kind regards,
M.
 
Old 08-04-2012, 07:54 AM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Quote:
Originally Posted by masuch View Post
I am fighting with rsyslog to find out why is it killed approximately each minute by ABRT signal. Could please anybody help me how to find what is causing it or/and why ?
Unfortunately not much to go on. FYI here's the Wikipedia description of SIGABRT and (asserting this is what you use) rsyslog Launchpad tickets (note Bug #1031962 "init: rsyslog main process killed by ABRT signal").

Questions:
- Did this happen due to an upgrade of Rsyslog or any of its dependencies? If so does reverting to the previous version work?
- Did this happen due to (re)configuration of /etc/rsyslog.conf? If so, what are the changes? And does a config check ("-f /path/to/configfile -N level") show everything is OK?
- Rsyslog allows for debug mode by starting it with the "-d" switch or sending SIGUSR1 to a running process. Does enabling debug mode show more nfo?
- Are there any other system changes that should be considered like software updates, AppArmor profiles or any other stuff?
* As a last resort, does starting Rsyslog from 'strace' ("-f -ff -o /path/to/logfile") show anything?
 
1 members found this post helpful.
Old 08-05-2012, 08:44 AM   #3
masuch
Member
 
Registered: Sep 2011
Location: /dev/null
Distribution: ubuntu 64bits
Posts: 135

Original Poster
Rep: Reputation: 1
Question

thanks a lot for information.

Quote:
note Bug #1031962 "init: rsyslog main process killed by ABRT signal"
bug was reported by me.

Quote:
- Did this happen due to an upgrade of Rsyslog or any of its dependencies? If so does reverting to the previous version work?
It did not happened by installation of rsyslog or any dependencies.
Quote:
- Did this happen due to (re)configuration of /etc/rsyslog.conf? If so, what are the changes? And does a config check ("-f /path/to/configfile -N level") show everything is OK?
errors:
Code:
rsyslogd -f /etc/rsyslog.conf -N 9
rsyslogd: version 5.8.6, config validation run (level 9), master config /etc/rsyslog.conf
rsyslogd: WARNING: rsyslogd is running in compatibility mode. Automatically generated config directives may interfer with your rsyslog.conf settings. We suggest upgrading your config and adding -c5 as the first rsyslogd option.
rsyslogd: the last error occured in /etc/rsyslog.d/relp.conf, line 2:"$InputRELPServerRun 20514"
rsyslogd: the last error occured in /etc/rsyslog.conf, line 71:"$IncludeConfig /etc/rsyslog.d/*.conf"
rsyslogd: CONFIG ERROR: could not interpret master config file '/etc/rsyslog.conf'. [try http://www.rsyslog.com/e/2124 ]
rsyslogd: Warning: backward compatibility layer added to following directive to rsyslog.conf: ModLoad immark
rsyslogd: Warning: backward compatibility layer added to following directive to rsyslog.conf: MarkMessagePeriod 1200
rsyslogd: Warning: backward compatibility layer added to following directive to rsyslog.conf: ModLoad imuxsock
but /etc/rsyslog.d/relp.conf is correct. do not know why config error happened ?
-c5 has been there.

Quote:
- Rsyslog allows for debug mode by starting it with the "-d" switch or sending SIGUSR1 to a running process. Does enabling debug mode show more nfo?
I have changed
RSYSLOGD_OPTIONS="-c5 -d" in /etc/default/rsyslog
but no any additional messages appeared so far.

Quote:
- Are there any other system changes that should be considered like software updates, AppArmor profiles or any other stuff?
I have added in AppArmor profile /etc/apparmor.d/usr.sbin.rsyslogd:
/var/rsyslog/** rw,

Quote:
* As a last resort, does starting Rsyslog from 'strace' ("-f -ff -o /path/to/logfile") show anything?
strace -f -ff -o rsyslog.strace > -ttt -T -s 100 rsyslogd -c5 -nd >rsyslog.debug
some errors:
1.
write(1, "open error 13, file '/var/rsyslog/work/dbq.00000001': Permission denied\n", 72) = 72 <0.000007>
-- I have created file manually with syslog:adm owner. seems working - some messages appeared there.
2.
write(1, "postgres query execution failed: PGRES_FATAL_ERROR\n", 51) = 51 <0.000759>
write(1, "7486.536509538:", 15) = 15 <0.000008>
write(1, "7f123cff9700: ", 14) = 14 <0.000007>
write(1, "pgsql, DBError(silent): db error (1): no connection to the server\n\n\n", 68) = 68 <0.000007>
...
stat("/home/syslog/.postgresql/postgresql.crt", 0x7f123cff5dd0) = -1 ENOENT (No such file or directory) <0.000009>
stat("/home/syslog/.postgresql/root.crt", 0x7f123cff5dd0) = -1 ENOENT (No such file or directory) <0.000009>
-- need to correct Postgre database connectivity and ... .
3.
write(1, "Called LogError, msg: db error (1146): Table 'Syslog.syslog_incoming' doesn't exist\n\n", 85) = 85 <0.000009>
-- has been corrected.
4.
write(1, "action 0x2078220 call returned -2007\n", 37) = 37 <0.000006>
write(1, "7439.534684181:", 15) = 15 <0.000006>
write(1, "7f1244fee700: ", 14) = 14 <0.000006>
write(1, "tryDoAction: unexpected error code -2007[nElem 1, Commited UpTo 0], finalizing\n", 79) = 79 <0.000006>
-- have no idea how to correct it ?
5.
rsyslogd: db error (1054): Unknown column 'invld' in 'field list'
-- have no idea how to correct it.
have found some info on http://kb.monitorware.com/100-cpu-ut...on-t10230.html
but apparently no solving at for long time.
I did not succeed to find the source causing this error.
6.
write(1, "cfline: '$ActionResumeRetryCount -1 # infinite retries on insert failure'\n", 75) = 75 <0.000005>
-- that should be correct because I have defined $ActionResumeRetryCount -1 so if no success that continue.
7.
open("/opt/oracle.instantclient_11_2.64b/libpq.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) <0.000006>
open("/opt/MonitorSoftware/lib/libpq.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) <0.000009>
open("/opt/AMDAPP/lib/x86_64/libpq.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) <0.000007>
open("/opt/AMDAPP/lib/x86/libpq.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) <0.000006>
-- created ln -s /usr/lib/libpq.so.5 libpq.so.5


killed by ABRT signal continues to appear :-(

Last edited by masuch; 08-05-2012 at 08:55 AM.
 
Old 08-05-2012, 09:40 AM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
I don't know what your /etc/rsyslog.d/*.conf and /etc/rsyslog.conf contain so I can't comment on or help you correct any of that. There's too much errors you can investigate and fix yourself before concluding Rsyslogd kills itself for some unknown reason.
 
Old 08-05-2012, 12:53 PM   #5
masuch
Member
 
Registered: Sep 2011
Location: /dev/null
Distribution: ubuntu 64bits
Posts: 135

Original Poster
Rep: Reputation: 1
Quote:
Originally Posted by unSpawn View Post
I don't know what your /etc/rsyslog.d/*.conf and /etc/rsyslog.conf contain so I can't comment on or help you correct any of that. There's too much errors you can investigate and fix yourself before concluding Rsyslogd kills itself for some unknown reason.
I am sorry. I forgot to show content.

/etc/rsyslog.conf:
Quote:
$ModLoad imuxsock
$ModLoad imklog
$ModLoad imudp
$UDPServerRun 1025 # I changed it from 514 because of "permission problems for port under 1024"
$ModLoad imtcp
$InputTCPServerRun 1025
$RepeatedMsgReduction on
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog
$WorkDirectory /var/spool/rsyslog
$SystemLogRateLimitInterval 5
$SystemLogRateLimitBurst 2500
*.* /var/log/everything.log
$IncludeConfig /etc/rsyslog.d/*.conf
/etc/rsyslog.d/postfix.conf:
Quote:
$AddUnixListenSocket /var/spool/postfix/dev/log
/etc/rsyslog.d/mysql.conf
Code:
$ModLoad ommysql
*.* :ommysql:localhost,Syslog,rsyslog,password
$template cacti_syslog,"insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
*.* >localhost,Syslog,rsyslog,password;cacti_syslog
$WorkDirectory /var/rsyslog/work
$ActionQueueType LinkedList 
$ActionQueueFileName dbq    
$ActionResumeRetryCount -1

/etc/rsyslog.d/pgsql.conf
Code:
$ModLoad ompgsql
*.* :ompgsql:127.0.0.1,Syslog,rsyslog,password
$template cacti_syslog,"insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
*.* >localhost,Syslog,rsyslog,password;cacti_syslog
/etc/rsyslog.d/relp.conf
Quote:
$ModLoad imrelp
$InputRELPServerRun 20514
/etc/rsyslog.d/10-octopussy.conf
Quote:
$WorkDirectory /var/lib/octopussy/rsyslog
$CreateDirs on
$MaxMessageSize 8k
$ActionQueueMaxDiskSpace 1g
$ActionQueueFileName rsyslog
$ActionQueueHighWaterMark 250000
$ActionQueueLowWaterMark 200000
$ActionQueueType LinkedList # [FixedArray/LinkedList/Direct/Disk]
$ActionQueueSaveOnShutdown on
$ActionQueueWorkerThreads 1 # 1 cpu
*.* |/var/spool/octopussy/octo_fifo
:hostname, !isequal, "myhostname" ~

/etc/rsyslog.d/20-ufw.conf
Quote:
:msg,contains,"[UFW " /var/log/ufw.log
& ~
/etc/rsyslog.d/50-default.conf
Code:
auth,authpriv.*			/var/log/auth.log
*.*;auth,authpriv.none	-/var/log/syslog
cron.*				      /var/log/cron.log
daemon.*			         -/var/log/daemon.log
kern.*				      -/var/log/kern.log
lpr.*				         -/var/log/lpr.log
user.*				      -/var/log/user.log
mail.*		-/var/log/mail.log
mail.info	-/var/log/mail.info
mail.warn	-/var/log/mail.warn
mail.err		/var/log/mail.err
news.crit		/var/log/news/news.crit
news.err			/var/log/news/news.err
news.notice		-/var/log/news/news.notice
*.=debug;\
	auth,authpriv.none;\
	news.none;mail.none	     -/var/log/debug
*.=info;*.=notice;*.=warn;\
	auth,authpriv.none;\
	cron,daemon.none;\
	mail,news.none		        -/var/log/messages
*.emerg                                :omusrmsg:*
daemon,mail.*;\
	news.=crit;news.=err;news.=notice;\
	*.=debug;*.=info;\
	*.=notice;*.=warn	/dev/tty8
daemon.*;mail.*;\
	news.err;\
	*.=debug;*.=info;\
	*.=notice;*.=warn	|/dev/xconsole

(At this moment I am fighting with rsyslogd: db error (1054): Unknown column 'invld' in 'field list'.
I understand that it is not safe to show to public my port mapping but I can change it any time and they should not be visible outside of my LAN network.
I do not need to push messages into mysql and postgre databases -just experimental - postgresql does not work yet.
As well I need to amend /dev/xconsole.
There should not be problem within latency of my SSD hard disk
(R/W IO operations are as fast as memory operations. I have this problem if I run the same OS instance cloned on SATA II/III hard disk/s.)


Please feel free to post any suggestions , recommendations , amendments, purging.
Thanks a lot for helping me.
 
Old 08-07-2012, 06:15 AM   #6
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Quote:
Originally Posted by masuch View Post
I am sorry. I forgot to show content.
Configs look OK to me but I'm no Rsyslogd guru.


Quote:
Originally Posted by masuch View Post
(At this moment I am fighting with rsyslogd: db error (1054): Unknown column 'invld' in 'field list'.
Last Adiscon thread I read the developers first suspicion was against MySQL insertion and later on in the same thread a suspicion was raised wrt (old school) Syslogd (not properly) defining or setting a wrong priority level that caused this.


Quote:
Originally Posted by masuch View Post
I do not need to push messages into mysql and postgre databases -just experimental - postgresql does not work yet.
I suggest you move the MySQL and PostGreSQL configs out of your /etc/rsyslog.d/ directory and see if that helps.


Quote:
Originally Posted by masuch View Post
As well I need to amend /dev/xconsole.
Please document changes and supply to distro or upstream?
 
Old 08-10-2012, 03:03 PM   #7
masuch
Member
 
Registered: Sep 2011
Location: /dev/null
Distribution: ubuntu 64bits
Posts: 135

Original Poster
Rep: Reputation: 1
Hi,

From that time I post something here
I compile new version 5.6.13 of rsyslog from git repository on recommendation to solve another problem - pthread_mutex_lock.c:62: __pthread_mutex_lock: Assertion `mutex->__data.__owner == 0' failed
(https://bugs.launchpad.net/ubuntu/+s...g/+bug/1024731)
- error did not appeared so far.

killed by ABRT signal did not appeared from that time either. I am not for 100 % sure what caused but seemed that problem was
due to missing chown syslog:adm /var/rsyslog/work (I have to more investigate)

/dev/xconsole. - I put
mknod -m 640 /dev/xconsole c 1 3
chown syslog:adm /dev/xconsole
in
/etc/init/rsyslog.conf
so problem with /dev/xconsole did not appeared again.

rsyslogd: db error (1054): Unknown column 'invld' in 'field list'.
did not solved yet.

I am still running in debug mode so I am now fighting with another problem which is logrotate - apparently does not work properly
within this configuration as it is confirmed on many other forums. - created my own logrotate running from crontab.

It is very difficult to read so huge files even for vim with file autoreloading functionality. Within 6 hours it has been generated 18Gbytes messages which was too much even for vim (I had to kill it by kill -9). sigterm did not work :-)

so ,basically this thread could be closed.

thanks a lot for helping me out.
M.
 
  


Reply

Tags
init, kernel, kill, rsyslog, signal


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Error Message "Process .usr/bin/nautilus was killed by signal 11 (SIGSEGV)" Linux_Nubie Linux - Newbie 4 02-29-2012 03:38 AM
init: ttyS0 main process terminated with status 1 linuxgentoo Ubuntu 1 07-29-2011 12:06 PM
kernel Panic init killed nidhichandyok Linux - Server 4 12-15-2009 07:26 AM
find history of a job killed by "kernel: Out of Memory: Killed process" poulacou Linux - Server 3 09-20-2007 05:24 PM
Process not killed with -KILL (-9) signal hondo Linux - General 2 07-25-2007 10:23 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 03:36 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration