/var/log/messages and /var/log/cron not working

sigkill · 08-07-2008, 12:00 PM

Hi, Im using CentOS 5. I don't know why my logs in /var/log/messages don't work including /var/log/cron
here's my previous logs in /var/log/messages

Code:

Aug  3 23:24:50 localhost smartd[2817]: smartd is exiting (exit status 0) 
Aug  3 23:24:50 localhost avahi-daemon[2701]: Got SIGTERM, quitting.
Aug  3 23:24:50 localhost avahi-daemon[2701]: Leaving mDNS multicast group on interface vmnet8.IPv6 with address fe80::250:56ff:fec0:8.
Aug  3 23:24:50 localhost avahi-daemon[2701]: Leaving mDNS multicast group on interface vmnet8.IPv4 with address 172.16.67.1.
Aug  3 23:24:50 localhost avahi-daemon[2701]: Leaving mDNS multicast group on interface vmnet1.IPv6 with address fe80::250:56ff:fec0:1.
Aug  3 23:24:50 localhost avahi-daemon[2701]: Leaving mDNS multicast group on interface vmnet1.IPv4 with address 172.16.123.1.
Aug  3 23:24:50 localhost avahi-daemon[2701]: Leaving mDNS multicast group on interface virbr0.IPv6 with address fe80::200:ff:fe00:0.
Aug  3 23:24:50 localhost avahi-daemon[2701]: Leaving mDNS multicast group on interface virbr0.IPv4 with address 192.168.122.1.
Aug  3 23:24:50 localhost dnsmasq[2699]: reading /etc/resolv.conf
Aug  3 23:24:50 localhost dnsmasq[2699]: using nameserver 192.168.1.1#53
Aug  3 23:24:50 localhost avahi-daemon[2701]: Leaving mDNS multicast group on interface eth0.IPv6 with address fe80::216:17ff:fe46:90d6.
Aug  3 23:24:50 localhost avahi-daemon[2701]: Leaving mDNS multicast group on interface eth0.IPv4 with address 192.168.0.183.
Aug  3 23:24:50 localhost libvirtd: Shutting down on signal 15
Aug  3 23:24:50 localhost dnsmasq[2699]: exiting on receipt of SIGTERM
Aug  3 23:24:58 localhost xinetd[7579]: Exiting...
Aug  3 23:25:02 localhost hcid[2252]: Got disconnected from the system message bus
Aug  3 23:25:02 localhost rpc.statd[2185]: Caught signal 15, un-registering and exiting.
Aug  3 23:25:02 localhost portmap[10752]: connect from 127.0.0.1 to unset(status): request from unprivileged port
Aug  3 23:25:03 localhost restorecond: terminated
Aug  3 23:25:03 localhost auditd[2093]: The audit daemon is exiting.
Aug  3 23:25:03 localhost kernel: audit(1217777103.290:237): audit_pid=0 old=2093 by auid=4294967295 subj=system_u:system_r:auditd_t:s0
Aug  3 23:25:03 localhost pcscd: pcscdaemon.c:572:signal_trap() Preparing for suicide
Aug  3 23:25:03 localhost kernel: usb 1-6: USB disconnect, address 2
Aug  3 23:25:03 localhost pcscd: hotplug_libusb.c:376:HPRescanUsbBus() Hotplug stopped
Aug  3 23:25:04 localhost pcscd: readerfactory.c:1379:RFCleanupReaders() entering cleaning function
Aug  3 23:25:04 localhost pcscd: pcscdaemon.c:532:at_exit() cleaning /var/run
Aug  3 23:25:04 localhost kernel: Kernel logging (proc) stopped.
Aug  3 23:25:04 localhost kernel: Kernel log daemon terminating.
Aug  3 23:25:05 localhost exiting on signal 15

any help please.Thanks!

unSpawn · 08-07-2008, 12:15 PM

Quote:

Originally Posted by sigkill

Aug 3 23:25:05 localhost exiting on signal 15

This only points to a halt or reboot, not syslog failure. Funny enough I vaguely recall a somwhat similar question not that long ago. When did this failure start? After some update? Does verifying the syslog package with RPM show its OK? When syslog is running does 'lsof -w -n -p $PID_of_syslog' or 'lsof -w -n +D /var' show files opened by syslog? Did anything change in (the (extended) access rights of) syslog.conf?

win32sux · 08-07-2008, 01:22 PM

Moved to Software.

sigkill · 08-08-2008, 08:53 AM

Quote:

When did this failure start? After some update?

I can remember, i was trying to get scripts work in crontab by using tail -f /valog/messages and /var/log/cron to see what's happening why my command in crontab is not working after a poweroff and started the next day, it's not receiving any logs.

Quote:

When syslog is running does 'lsof -w -n -p $PID_of_syslog' or 'lsof -w -n +D /var' show files opened by syslog?

Code:

[root@localhost ~]# lsof -w -n -p 2045
COMMAND  PID USER   FD   TYPE DEVICE    SIZE     NODE NAME
syslogd 2045 root  cwd    DIR    8,2    4096        2 /
syslogd 2045 root  rtd    DIR    8,2    4096        2 /
syslogd 2045 root  txt    REG    8,2   35832 14633316 /sbin/syslogd
syslogd 2045 root  mem    REG    8,2  125736 11001562 /lib/ld-2.5.so
syslogd 2045 root  mem    REG    8,2 1597968 11001577 /lib/libc-2.5.so
syslogd 2045 root  mem    REG    8,2   46680 10999336 /lib/libnss_files-2.5.so

Quote:

Did anything change in (the (extended) access rights of) syslog.conf?

Code:

[root@localhost ~]# ls -l /etc/syslog.conf 
-rw-r--r-- 1 root root 694 May 25 08:45 /etc/syslog.conf

please let me know if theres anything wrong with all of them of if i executed the right command you were asking.

heres and ls -l of my /var/log

Code:

-rw-r----- 1 root root   10034 Aug  8 20:35 acpid
-rw------- 1 root root  483065 Jul 22 05:19 anaconda.log
-rw------- 1 root root   18995 Jul 22 05:19 anaconda.syslog
-rw------- 1 root root   56454 Jul 22 05:19 anaconda.xlog
drwxr-x--- 2 root root    4096 May 25 14:28 audit
-rw------- 1 root root     927 Aug  3 17:44 boot.log
-rw------- 1 root root     258 Aug  3 14:22 boot.log.1
-rw------- 1 root root     363 Jul 26 18:54 boot.log.2
-rw------- 1 root utmp    1536 Aug  3 19:36 btmp
drwxr-xr-x 2 root root    4096 Nov 12  2007 conman
drwxr-xr-x 2 root root    4096 Nov 12  2007 conman.old
-rw------- 1 root root    4207 Aug  3 23:03 cron
-rw------- 1 root root    6149 Aug  3 15:27 cron.1
-rw------- 1 root root    5365 Jul 28 06:12 cron.2
drwxr-xr-x 2 lp   sys     4096 Aug  3 15:27 cups
-rw-r--r-- 1 root root    1107 Aug  8 20:35 dkms_autoinstaller
-rw-r--r-- 1 root root   17835 Aug  8 20:34 dmesg
-rw------- 1 root root   12096 Jul 25 04:29 faillog
drwxr-xr-x 2 root root    4096 Aug  8 20:35 gdm
drwx------ 2 root root    4096 Jan 16  2008 httpd
-rw-r--r-- 1 root root  147168 Aug  8 20:35 lastlog
drwxr-xr-x 3 root root    4096 Aug  3 16:16 libvirt
drwxr-xr-x 2 root root    4096 Jul 22 05:12 mail
-rw------- 1 root root   10395 Aug  3 22:20 maillog
-rw------- 1 root root    9788 Aug  3 15:27 maillog.1
-rw------- 1 root root    8554 Jul 28 06:12 maillog.2
-rw-r----- 1 root root   90136 Aug  3 23:25 messages
-rw------- 1 root root 3142567 Aug  3 15:25 messages.1
-rw------- 1 root root 1915684 Jul 28 05:36 messages.2
drwxr-xr-x 2 root root    4096 Jul 22 08:14 pm
drwx------ 2 root root    4096 Mar 15  2007 ppp
drwxr-xr-x 2 root root    4096 Jul 22 20:21 prelink
-rw-r--r-- 1 root root   27917 Aug  8 21:41 rpmpkgs
-rw-r--r-- 1 root root   27419 Aug  2 12:11 rpmpkgs.1
-rw-r--r-- 1 root root   27352 Jul 24 22:09 rpmpkgs.2
drwx------ 2 root root    4096 Jun 22 09:05 samba
-rw-r--r-- 1 root root  131359 Jul 22 20:21 scrollkeeper.log
-rw------- 1 root root    2335 Aug  3 23:24 secure
-rw------- 1 root root    8799 Aug  3 15:11 secure.1
-rw------- 1 root root   24373 Jul 28 05:35 secure.2
-rw------- 1 root root       0 Aug  3 15:27 spooler
-rw------- 1 root root       0 Jul 28 06:12 spooler.1
-rw------- 1 root root       0 Jul 22 05:12 spooler.2
-rw------- 1 root root       0 Jul 22 08:05 tallylog
drwxr-xr-x 2 root root    4096 May 24 23:06 vbox
drwxr-xr-x 3 root root    4096 Aug  8 21:31 vmware
-rw-rw-r-- 1 root utmp  460032 Aug  8 21:24 wtmp
drwx------ 2 root root    4096 Aug  3 16:38 xen
-rw-r--r-- 1 root root   62062 Aug  8 20:35 Xorg.0.log
-rw-r--r-- 1 root root   63135 Aug  8 19:59 Xorg.0.log.old
-rw-r--r-- 1 root root   20841 Aug  7 16:36 yum.log

Ive noticed that that all of the files that don't have the read permission were also the same files that stoped logging according to it's date "Aug 3". Does it affect it?

unSpawn · 08-08-2008, 11:07 AM

There's a few ways to get more info. One could be to search Linuxquestions.org and the CentOS bug tracker, mailing lists and forum for similar problems. If that doesn't yield anything expand search to Red Hat and other OS sources. If that doesn't yield anything then you've covered everything that could be faster and more efficient and we'll try to work it out ourselves. BTW, does klogd run OK? Your syslogd PID 2045 shows it hasn't opened /dev/log. As root account user, notice the commandline of your running syslog ('pgrep -lf syslogd'), kill it (don't use '/etc/init.d/syslogd stop': keep klogd running), then try to start syslogd from the commandline as '/usr/bin/strace -v -o /tmp/syslog.strace /sbin/syslogd', adding these: "-f /etc/syslog.conf -d -a /dev/log 2>&1 | tee /tmp/syslog.tee" to your default switches to force it to read syslog.conf, enter debug mode and stay in the foreground, force it to use /dev/log and copy stdout and stderr to the file /tmp/syslog.tee. In another terminal screen execute 'pkill -USR1 -f /sbin/syslogd' to make it spit out debug messages if it doesn't already, then type 'logger PING', then type '( \ps axZ|grep syslogd; \ls -alZ /etc/syslog.conf /dev/log /sbin/syslogd; rpm -qVv sysklogd; /usr/sbin/lsof -w -n +D /var/log; \ls -aldZ / /var /var/log /var/log/messages; ) | tee /tmp/syslog.attr'. Now switch back to the first terminal window and CTRL+C to kill strace, then '/etc/init.d/syslogd restart'. Now you have three logs: /tmp/syslog.{tee,strace,attr} to read.

If reading those logs doesn't work for you (and I guess it's a wee bit too much lines to post here?) please upload logs as tarball to some free hoster and post the URI here. Before doing so replace any information in your logs if you need to but please don't delete lines unless you know for certain it won't affect log diagnosis.

sigkill · 08-09-2008, 04:30 AM

I solved the problem by installing rsyslog, don't know yet what will happen next. my primary concern is to get log files working. Thanks for response and tips.

unSpawn · 08-09-2008, 02:08 PM

Quote:

Originally Posted by sigkill

I solved the problem by installing rsyslog

isn't that avoiding the problem? I mean what happens if rsyslog fails you too?...
Anyway it's good to see you got it working.