The problem, in short, is that /var/log/messages is no longer being populated with records on my Fedora Core 6 box. Neither is /var/log/secure. This has happened to me twice. The first time I chose not to take any chances; I wiped the box and re-installed Core 6 again. Now that it has happened a second time I'm more interested in finding the problem.

The only application on the box that I can think of that has a large impact on the system would be vmware. I just don't know where to start investigating the problem. Here are the logs from the last time I apparently shut down the machine:

Mar 6 15:42:00 nss-66 kernel: device eth0 left promiscuous mode
Mar 6 15:42:00 nss-66 kernel: audit(1173213720.677:105): dev=eth0 prom=0 old_prom=256 auid=4294967295
Mar 6 15:42:00 nss-66 kernel: bridge-eth0: disabled promiscuous mode
Mar 6 15:42:00 nss-66 kernel: device eth0 entered promiscuous mode
Mar 6 15:42:00 nss-66 kernel: audit(1173213720.677:106): dev=eth0 prom=256 old_prom=0 auid=4294967295
Mar 6 15:42:00 nss-66 kernel: bridge-eth0: enabled promiscuous mode
Mar 6 15:42:01 nss-66 kernel: device eth0 left promiscuous mode
Mar 6 15:42:01 nss-66 kernel: audit(1173213721.280:107): dev=eth0 prom=0 old_prom=256 auid=4294967295
Mar 6 15:42:01 nss-66 kernel: bridge-eth0: disabled promiscuous mode
Mar 6 15:42:10 nss-66 gconfd (maher-12644): Exiting
Mar 6 15:42:12 nss-66 gdm[3322]: Master halting...
Mar 6 15:42:12 nss-66 escd: *** glibc detected *** ./escd: free(): invalid pointer: 0xbfda6684 ***
Mar 6 15:42:14 nss-66 shutdown[3322]: shutting down for system halt
Mar 6 15:42:16 nss-66 smartd[3268]: smartd received signal 15: Terminated
Mar 6 15:42:16 nss-66 smartd[3268]: smartd is exiting (exit status 0)
Mar 6 15:42:16 nss-66 avahi-daemon[3186]: Got SIGTERM, quitting.
Mar 6 15:42:16 nss-66 avahi-daemon[3186]: Leaving mDNS multicast group on interface vmnet1.IPv6 with address fe80::250:56ff:fec0:1.
Mar 6 15:42:16 nss-66 avahi-daemon[3186]: Leaving mDNS multicast group on interface vmnet1.IPv4 with address x.x.x.x.
Mar 6 15:42:16 nss-66 avahi-daemon[3186]: Leaving mDNS multicast group on interface vmnet8.IPv6 with address x:x:x:x:x:x.
Mar 6 15:42:16 nss-66 avahi-daemon[3186]: Leaving mDNS multicast group on interface vmnet8.IPv4 with address x.x.x.x.
Mar 6 15:42:16 nss-66 avahi-daemon[3186]: Leaving mDNS multicast group on interface eth0.IPv6 with address x:x:x:x:x:x.
Mar 6 15:42:16 nss-66 avahi-daemon[3186]: Leaving mDNS multicast group on interface eth0.IPv4 with address x.x.x.x.
Mar 6 15:42:27 nss-66 xinetd[22442]: Exiting...
Mar 6 15:42:31 nss-66 hcid[2669]: Got disconnected from the system message bus
Mar 6 15:42:31 nss-66 rpc.statd[2499]: Caught signal 15, un-registering and exiting.
Mar 6 15:42:31 nss-66 pcscd: pcscdaemon.c:529:signal_trap() Preparing for suicide
Mar 6 15:42:32 nss-66 pcscd: hotplug_libusb.c:361:HPEstablishUSBNotifications() Hotplug stopped
Mar 6 15:42:32 nss-66 pcscd: readerfactory.c:1350:RFCleanupReaders() entering cleaning function
Mar 6 15:42:32 nss-66 pcscd: pcscdaemon.c:489:at_exit() cleaning /var/run
Mar 6 15:42:32 nss-66 kernel: Kernel logging (proc) stopped.
Mar 6 15:42:32 nss-66 kernel: Kernel log daemon terminating.
Mar 6 15:42:34 nss-66 exiting on signal 15

Any help would be greatly appreciated.

John

What's in /etc/syslog.conf? I'm not sure whether Fedora uses anything more sophisticated, but that's the default config file for the syslogd daemon that writes the log files. It's a place to start.

Thanks for responding. /etc/syslog,conf seems pretty typical. Here it is:

*.info;mail.none;authpriv.none;cron.none /var/log/messages
authpriv.* /var/log/secure
mail.* -/var/log/maillog
cron.* /var/log/cron
*.emerg *
uucp,news.crit /var/log/spooler
local7.* /var/log/boot.log

John

Maybe something is killing the syslog daemon off when it's not supposed to be. Try testing with logger and see what happens.

Ex:

logger -p user.info -t log_test -- "Logging with USER/INFO"

logger -p local2.notice -t log_test -- "Logging with LOCAL2/NOTICE"


See if those appear as expected. If so, then probably something is shutting syslog down, such as log rotate, some sort of maint. cron job, etc. Make sure the logfile itself is R/W-able.


BTW, this line has me curious. This is behavior consistant with a sniffer:

Someone running tcpdump, Wireshark, or ngrep would make such an entry.

Good idea. Unfortunately, it produced no logs in /var/log/messages. And yet, syslog is still running:

# service syslog status
syslogd (pid 2343) is running...
klogd (pid 2346) is running...

The files are all r/w-able.

That had me currious as well, but I think vmware does that when bridging is used to give the vmware guest OS access to the NIC. I, of course, could be mistaken.

I do appreciate your suggestions. Any other thoughts? I'm getting ready to re-install my system again.

John

Only a few other things I can think of. None of them are solid answers, just leads. One would be syslogd opening the "wrong" logfiles after logrotation. To see run "lsof -w -n +D /var/log" and grep for the logfile names or pids. Sending the PID's a HUP signal should fix that. Another check could be for read-only mounts, but that only works if /var(/log) resides on its own partition which got remounted read-only (on purpose by the kernel) after encountering FS errors (check with "mount" or "touch" a file). Another check could be if SELinux is enabled and if some whack rule prevents syslogd from writing. Running the system with SELinux disabled should show. Very unlikely though. Other things to check with could be temporarily running an "older" (with SELinux off), verifying contents of the Syslogd package and running audit checks from a Live CD, wouldn't hurt to do it anyway, you can always use the practice to your advantage later on...

This was very educational. As expected, /var/log/messages, /var/log/secure, and other files were not open. Restarting the syslog service changed nothing.

Bingo! I changed SELinux to Permissive, rebooted (necessary?), and shazam! Logs, logs, logs. But why? I suspect it was related to some update since I installed the OS. I never messed with SELinux rules (I essentially know nothing about SELinux). So, why would an update (if that's the culprit) effectively disable logging? Sounds like a big problem.

How would one run and audit check from a Live CD?

Thanks very much for your help.

John

I 've been having the same EXACT problems as jmaher. I've been examining the problem for 2 days off and on when I have time. Setting /etc/selinux/config to permissive does indeed resolve the issue. It was enforcing.

2 Things to note of consequence. 1) My logs stopped on March 10th, I run updates manualy so the issue has to be with an update prior to March 10th and no recent patch has resolved the issue.
2) My installation is x86_64 with vmware installed. I am fairly certain that I installed Selinux in permissive mode. One of the patches perhaps moved it to enforcing?

In any scenario I believe the devs should look at selinux and consider "enforcing" mode blocking messages and security not only a bug but security risk

Thanks for the responses above. Much appreciated

Can both of you check /var/log/yum.log for any selinux updates around the timeframe that logging broke. There were several updates in March, which makes me wonder if one happened to break stuff. If there weren't any selinux updates at that time, could you post a list of all rpms updated around that period.

As requested see below:
[root@(security by obscurity :-)) log]# less yum.log.1 |grep selinu
Feb 03 16:42:51 Updated: libselinux.x86_64 1.33.4-2.fc6
Feb 03 16:43:03 Updated: libselinux.i386 1.33.4-2.fc6
Feb 03 16:43:56 Updated: libselinux-python.x86_64 1.33.4-2.fc6
Feb 03 16:47:10 Updated: selinux-policy.noarch 2.4.6-27.fc6
Feb 03 16:48:12 Updated: selinux-policy-targeted.noarch 2.4.6-27.fc6
Feb 11 10:40:00 Updated: selinux-policy.noarch 2.4.6-35.fc6
Feb 11 10:40:19 Updated: selinux-policy-targeted.noarch 2.4.6-35.fc6
Feb 25 12:24:42 Updated: selinux-policy.noarch 2.4.6-40.fc6
Feb 25 12:26:09 Updated: selinux-policy-targeted.noarch 2.4.6-40.fc6
Mar 08 19:51:07 Updated: selinux-policy.noarch 2.4.6-41.fc6
Mar 08 19:51:15 Updated: selinux-policy-targeted.noarch 2.4.6-41.fc6
Mar 23 13:54:06 Updated: selinux-policy.noarch 2.4.6-42.fc6
Mar 23 13:54:47 Updated: selinux-policy-targeted.noarch 2.4.6-42.fc6
Apr 02 14:15:46 Updated: selinux-policy.noarch 2.4.6-46.fc6
Apr 02 14:16:21 Updated: selinux-policy-targeted.noarch 2.4.6-46.fc6
[root@(edited for our younger viewers:-) log]#

I suspect that one of the packages in bold is where to start looking.
as much as I am curious about selinux and spent a cpl of hours looking at docs; - to date in production as well as home environments, I've found it to be more of a hinderance than a help imho. Post up if you need more info i'd be happy to provide it. I rotated my logs(after inspecting them) just prior to finding this thread trying to trouble shoot the issue, so essentially I have no real logs left.

BM:-)

I would do a wee test by temporarily downgrading your SELinux policy to see if logging works again. This could help isolate the cause, after which you should fire off a bug report.

SELinux is a real security enhancement and I hope it's here to stay. RH/FC/the community really put in a lot of effort to make it (relatively) easier to use (check the differences between FC5 and late FC6 policy management to see what I mean). If you can submit a bug report you will help improve it, which makes it a smoother ride for everyone.

My logs (on this particular FC 6 machine) apparently broke before March 4, but I can't be more specific, so here are all of the selinux updates within my yum.log:

Dec 04 20:00:55 Updated: selinux-policy.noarch 2.4.6-1.fc6
Dec 04 20:02:57 Updated: selinux-policy-targeted.noarch 2.4.6-1.fc6
Dec 16 05:35:45 Updated: libselinux.i386 1.33.2-3.fc6
Dec 16 05:36:28 Updated: libselinux-python.i386 1.33.2-3.fc6
Dec 16 05:36:47 Updated: selinux-policy.noarch 2.4.6-7.fc6
Dec 16 05:37:35 Updated: libselinux-devel.i386 1.33.2-3.fc6
Dec 16 05:38:25 Updated: selinux-policy-targeted.noarch 2.4.6-7.fc6
Dec 20 20:43:54 Updated: selinux-policy.noarch 2.4.6-13.fc6
Dec 20 20:44:27 Updated: selinux-policy-targeted.noarch 2.4.6-13.fc6
Jan 07 22:50:08 Updated: selinux-policy.noarch 2.4.6-17.fc6
Jan 07 22:50:41 Updated: selinux-policy-targeted.noarch 2.4.6-17.fc6
Jan 15 17:56:06 Updated: libselinux.i386 1.33.3-2.fc6
Jan 15 17:57:42 Updated: selinux-policy.noarch 2.4.6-23.fc6
Jan 15 17:57:56 Updated: libselinux-python.i386 1.33.3-2.fc6
Jan 15 17:58:13 Updated: selinux-policy-targeted.noarch 2.4.6-23.fc6
Jan 15 17:58:25 Updated: libselinux-devel.i386 1.33.3-2.fc6
Jan 18 20:39:57 Updated: libselinux.i386 1.33.4-2.fc6
Jan 18 20:41:26 Updated: libselinux-python.i386 1.33.4-2.fc6
Jan 18 20:42:10 Updated: libselinux-devel.i386 1.33.4-2.fc6
Jan 23 20:50:07 Updated: selinux-policy.noarch 2.4.6-27.fc6
Jan 23 20:56:53 Updated: selinux-policy-targeted.noarch 2.4.6-27.fc6
Feb 07 21:38:12 Updated: selinux-policy.noarch 2.4.6-35.fc6
Feb 07 21:39:35 Updated: selinux-policy-targeted.noarch 2.4.6-35.fc6
Feb 26 21:46:02 Updated: selinux-policy.noarch 2.4.6-40.fc6
Feb 26 21:47:15 Updated: selinux-policy-targeted.noarch 2.4.6-40.fc6
Mar 03 08:37:32 Updated: selinux-policy.noarch 2.4.6-41.fc6
Mar 03 08:39:39 Updated: selinux-policy-targeted.noarch 2.4.6-41.fc6
Mar 12 23:11:01 Updated: selinux-policy.noarch 2.4.6-42.fc6
Mar 12 23:11:18 Updated: selinux-policy-targeted.noarch 2.4.6-42.fc6
Mar 28 16:37:31 Updated: selinux-policy.noarch 2.4.6-46.fc6
Mar 28 16:38:10 Updated: selinux-policy-targeted.noarch 2.4.6-46.fc6

I had a similar problem with FC6...no logs to /var/log/messages or /var/log/secure and possibly others.

I resolved the issue by changing the SELinux booleans for syslogd and klogd; then restarting syslogd (restarting the daemon may be unnecessary, but I did it anyway).

// modify selinux policy
setsebool -P syslogd_disable_trans=1
setsebool -P klogd_disable_trans=1

// ensure selinux enforcing
setenforce 1

// restart syslogd
service syslog restart

// test with logger
logger test

// check /var/log/messages