/var/log/messages no longer being written to

Mehuge · 06-27-2006, 11:46 AM

After a reboot the other day /var/log/messages stopped being written to. I possibly had a new kernel version waiting to kick in from a yum update, so it may be related to a new kernel.

Kernel 2.6.16-1.2096_FC4smp

In fact, looking in /var/log ... none of the following logs are being written to:

4 -rw------- 1 root root 0 Jun 25 04:30 spooler
4 -rw------- 1 root root 0 Jun 25 04:30 secure
4 -rw------- 1 root root 0 Jun 25 04:30 messages
4 -rw------- 1 root root 0 Jun 25 04:30 maillog
4 -rw------- 1 root root 0 Jun 25 04:30 cron
4 -rw------- 1 root root 0 Jun 25 04:30 boot.log
4 -rw------- 1 root root 0 Jun 25 04:30 xferlog
4 -rw-r--r-- 1 root root 0 Jun 25 04:30 up2date

/etc/syslog.conf contains

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
kern.* /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;news.none;authpriv.none;cron.none /var/log/messages

# The authpriv file has restricted access.
authpriv.* /var/log/secure

# Log all the mail messages in one place.
mail.* -/var/log/maillog

# Log cron stuff
cron.* /var/log/cron

# Everybody gets emergency messages
*.emerg *

# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler

# Save boot messages also to boot.log
local7.* /var/log/boot.log

#
# INN
#
news.=crit /var/log/news/news.crit
news.=err /var/log/news/news.err
news.notice /var/log/news/news.notice

# Catch user notices
user.* /var/log/user.notice.log

I did a service syslog restart and it did show logger as not running, and started it up, but still I get nothing in any of my logs.

Any ideas how I can find out what is going on?

unSpawn · 06-27-2006, 12:27 PM

As root, verify klogd is running: pgrep -lf klogd
then get the PID of syslogd and check if it has the files open:
lsof -p $(pgrep -f syslog) -a +D /var/log
If none, see what "dmesg" has got to say and check available diskspace with "df".
If no problem, reboot the old kernel and see if that still works.

Mehuge · 06-28-2006, 02:39 AM

Still no joy. Outputs from the commands you listed:

[root@hive log]# pgrep -lf klogd
21461 klogd -x -c 4

[root@hive log]# pgrep -lf syslogd
21459 syslogd -m 0

[root@hive log]# lsof -p $(pgrep -f syslog) -a +D /var/log

[root@hive log]# lsof -p $(pgrep -f syslogd)
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
syslogd 21459 root cwd DIR 253,0 4096 2 /
syslogd 21459 root rtd DIR 253,0 4096 2 /
syslogd 21459 root txt REG 253,0 35696 14829486 /sbin/syslogd
syslogd 21459 root mem REG 253,0 1481808 16859053 /lib/libc-2.3.6.so
syslogd 21459 root mem REG 253,0 46640 16859091 /lib/libnss_files-2.3.6.so
syslogd 21459 root mem REG 0,0 0 [vdso] (stat: No such file or directory)
syslogd 21459 root mem REG 253,0 126576 16859051 /lib/ld-2.3.6.so

[root@hive log]# df /var/log
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/vg00-lv00
73545144 59080680 10668292 85% /

[root@hive log]# dmesg | grep syslog
audit(1151150355.645:2): avc: denied { read } for pid=2204 comm="syslogd" nam
e="services" dev=dm-0 ino=14567535 scontext=system_u:system_r:syslogd_t tcontext
=root

bject_r:tmp_t tclass=file

[root@hive log]# ps -p 2204
PID TTY TIME CMD

I rebooted using the previous kernel (2.6.15-1.1833_FC4smp) and it made no difference.

I watched the boot process and nothing significant failed suring bootup (vmware and nagios was complaining about a lock file).

Starting system logger [OK]
Starting kernel logger [OK]

Going to switch back to the normal kernel.

Mehuge · 06-28-2006, 02:59 AM

Humm, strange. I can get logging to work if I manually run klogd & syslogd. I did the following:

[root@hive]# service stop syslog
Stopping kernel logger [OK]
Stopping system logger [OK]
[root@hive]# klogd -d
[root@hive]# syslogd -d

But if I kill those processes off and start it via the service command again, it doenst work.

[root@hive init.d]# ls -l syslog
-rwxr-xr-x 1 root root 1369 May 19 2005 syslog

From /etc/sysconfig/syslog
# Options to syslogd
# -m 0 disables 'MARK' messages.
# -r enables logging from remote machines
# -x disables DNS lookups on messages recieved with -r
# See syslogd(8) for more details
SYSLOGD_OPTIONS="-m 0"
# Options to klogd
# -2 prints all kernel oops messages twice; once for klogd to decode, and
# once for processing with 'ksymoops'
# -x disables all klogd processing of oops messages entirely
# See klogd(8) for more details
KLOGD_OPTIONS="-x -c 4"

I edited /etc/init.d/functions and added a set -x and got the following output:

Starting system logger: + local gotbase= force=
+ local base= user= nice= bg= pid=
+ nicelevel=0
+ '[' syslogd '!=' syslogd ']'
+ '[' -z '' ']'
+ base=syslogd
+ '[' -f /var/run/syslogd.pid ']'
+ '[' -n '' -a -z '' ']'
+ ulimit -S -c 0
+ '[' -n '' ']'
+ '[' color = verbose -a -z '' ']'
+ '[' -z '' ']'
+ syslogd -m 0
+ '[' 0 -eq 0 ']'
+ success 'syslogd startup'
+ '[' color '!=' verbose -a -z '' ']'
+ echo_success
+ '[' color = color ']'
+ echo -en '\033[60G'
+ echo -n '[ '
[ + '[' color = color ']'
+ echo -en '\033[0;32m'
+ echo -n OK
OK+ '[' color = color ']'
+ echo -en '\033[0;39m'
+ echo -n ' ]'
]+ echo -ne '\r'
+ return 0
+ return 0
+ RETVAL=0

I tried running the commands from above manually, and it works:

[root@hive init.d]# klogd -x -c 4
[root@hive init.d]# syslogd -m 0

But if I shutdown and restart via the service command, it doesnt work.

[root@hive init.d]# service syslog restart
Shutting down kernel logger: [ OK ]
Shutting down system logger: [ OK ]
Starting system logger: [ OK ]
Starting kernel logger: [ OK ]

The last thing I see in /var/log/messages is:

Jun 28 09:15:29 hive kernel: Kernel logging (proc) stopped.
Jun 28 09:15:29 hive kernel: Kernel log daemon terminating.
Jun 28 09:15:30 hive exiting on signal 15

Only difference is the ulimit commands.

Mehuge · 06-28-2006, 04:12 AM

The ulimit does not seem to be the issue either, I wrote this test script:

Code:

#!/bin/bash
service syslog stop
ulimit -S -c 0
syslogd -m 0
klogd -x -c 4
lsof -p $(pgrep syslog)
tail -5 /var/log/messages
service syslog restart
tail -5 /var/log/messages
lsof -p $(pgrep syslog)

Which produced the following results:

Code:

[root@hive log]# bash -x /tmp/syslog
+ service syslog stop
Shutting down kernel logger: [OK]
Shutting down system logger: [OK]
+ ulimit -S -c 0
+ syslogd -m 0
+ klogd -x -c 4
++ pgrep syslog
+ lsof -p 10592
COMMAND   PID USER   FD   TYPE     DEVICE    SIZE     NODE NAME
syslogd 10592 root  cwd    DIR      253,0    4096        2 /
syslogd 10592 root  rtd    DIR      253,0    4096        2 /
syslogd 10592 root  txt    REG      253,0   35696 14829486 /sbin/syslogd
syslogd 10592 root  mem    REG      253,0   46640 16859091 /lib/libnss_files-2.3.6.so
syslogd 10592 root  mem    REG      253,0 1481808 16859053 /lib/libc-2.3.6.so
syslogd 10592 root  mem    REG        0,0                0 [vdso] (stat: No such file or directory)
syslogd 10592 root  mem    REG      253,0  126576 16859051 /lib/ld-2.3.6.so
syslogd 10592 root    0u  unix 0xf6964780            65338 /dev/log
syslogd 10592 root    1w   CHR        5,1             1847 /dev/console
syslogd 10592 root    2w   REG      253,0    5008 12439731 /var/log/messages
syslogd 10592 root    3w   REG      253,0   56486 12440898 /var/log/secure
syslogd 10592 root    4w   REG      253,0       0 12440919 /var/log/maillog
syslogd 10592 root    5w   REG      253,0    1142 12440914 /var/log/cron
syslogd 10592 root    6w   REG      253,0       0 12441615 /var/log/spooler
syslogd 10592 root    7w   REG      253,0       0 12441635 /var/log/boot.log
syslogd 10592 root    8w   REG      253,0  852024 12473620 /var/log/news/news.crit
syslogd 10592 root    9w   REG      253,0  575791 12473621 /var/log/news/news.err
syslogd 10592 root   10w   REG      253,0 3466481 12473619 /var/log/news/news.notice
syslogd 10592 root   11w   REG      253,0 8061764 12440841 /var/log/user.notice.log
+ tail -5 /var/log/messages
Jun 28 09:57:10 hive kernel: Kernel logging (proc) stopped.
Jun 28 09:57:10 hive kernel: Kernel log daemon terminating.
Jun 28 09:57:11 hive exiting on signal 15
Jun 28 10:06:00 hive syslogd 1.4.1: restart.
Jun 28 10:06:00 hive kernel: klogd 1.4.1, log source = /proc/kmsg started.
+ service syslog restart
Shutting down kernel logger: [OK]
Shutting down system logger: [OK]
Starting system logger: [OK]
Starting kernel logger: [OK]
+ tail -5 /var/log/messages
Jun 28 10:06:00 hive syslogd 1.4.1: restart.
Jun 28 10:06:00 hive kernel: klogd 1.4.1, log source = /proc/kmsg started.
Jun 28 10:06:00 hive kernel: Kernel logging (proc) stopped.
Jun 28 10:06:00 hive kernel: Kernel log daemon terminating.
Jun 28 10:06:01 hive exiting on signal 15
++ pgrep syslog
+ lsof -p 10652
COMMAND   PID USER   FD   TYPE DEVICE    SIZE     NODE NAME
syslogd 10652 root  cwd    DIR  253,0    4096        2 /
syslogd 10652 root  rtd    DIR  253,0    4096        2 /
syslogd 10652 root  txt    REG  253,0   35696 14829486 /sbin/syslogd
syslogd 10652 root  mem    REG  253,0   46640 16859091 /lib/libnss_files-2.3.6.so
syslogd 10652 root  mem    REG  253,0 1481808 16859053 /lib/libc-2.3.6.so
syslogd 10652 root  mem    REG    0,0                0 [vdso] (stat: No such file or directory)
syslogd 10652 root  mem    REG  253,0  126576 16859051 /lib/ld-2.3.6.so

As this shows, when started directly it works, but when started through the service script it does not.

Im at a bit of a loss what to try next to get this working. I do now have a partial workaround, which will be to start it manually, but im loosing boot-time messages because it doesnt work during bootup.

unSpawn · 06-28-2006, 04:25 AM

Hold on... you're too fast replying to yourself :-]

[root@hive log]# dmesg | grep syslog
audit(1151150355.645:2): avc: denied { read } for pid=2204 comm="syslogd" name="services" dev=dm-0 ino=14567535 scontext=system_u:system_r:syslogd_t tcontext=rootbject_r:tmp_t tclass=file
You have SELinux enabled. Try running "fixfiles relabel" instead (or audit2allow if you know how to deal with local SELinux rules).

Mehuge · 06-28-2006, 07:33 AM

unSpawn, your a star, or at least I think you might be.

I ran fixfiles relabel and now syslog is working when started via the services script. I did also do a yum update but dont think this updated anything that was relevent to the syslog problem. Not quite sure what it did that fixed it, but the following were changed that were syslog related:

Jun 28 12:07:21 hive setfiles: relabeling /var/run/syslogd.pid from root

bject_
r:sendmail_var_run_t to system_u

bject_r:syslogd_var_run_t
Jun 28 12:07:21 hive setfiles: relabeling /var/run/klogd.pid from root

bject_r:
sendmail_var_run_t to system_u

bject_r:klogd_var_run_t

It didnt seem to touch /var/log/messages or any of the ones syslogd is setup to write too.

Strange, but thanks for your input.

mmyjak · 01-31-2008, 11:06 AM

I seem to have this very same problem - syslog will not start correctly from /etc/rc.d/init.d/syslog start but a manual start from /sbin/syslog appears to work and output to /var/log/messages. Space is fine (33% used on /var) and all else appears to be normal.

But after a reboot, syslog fails and there's no output. And restarting with syslog -m 0 from the init file returns [failed], even though the process remains up and running. Sending it a kill -SIGHUP `cat /

I'm running an older version (redhat 6.1) and fixfiles is not a working function in kernel 2.2.

ideas?
-M

chrism01 · 02-04-2008, 10:58 PM

If you're truly running RH6.1 with kernel 2.2 (which is poss), that's so old it's antedeluvian

Seriously, that hasn't been supported in forever.
If you like RH, try Fedora 8 (kernel 2.6) or the Centos series (which is the RH Enterprise free copy).