syslogd running but not logging on boot

Dark Typhoon · 03-20-2009, 01:11 PM

I recently upgraded the Perl implementation (in addition to a number of other components) on a RHEL 4 system. I also made modifications to the /etc/syslog.conf and /etc/sysconfig/syslog files. On a reboot, boot the syslogd and klogd daemon come up as evidenced here:

Code:

$ ps ax | grep logd | grep -v mini | grep -v grep
 3166 ?        Ss     0:00 syslogd -m 0 -x -r
 3170 ?        Ss     0:00 klogd -x

However, no messages are logged to /var/log/messages

If I log in as root and stop the daemon via

Code:

$ /etc/init.d/syslog stop

then restart them manually as illustrated below, the daemons come up and start logging properly.

Code:

$ syslogd -m 0 -x -r
$ klogd -x

Here's my /etc/syslog.conf file:

Code:

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
# scratch that, log everything!!
*.*                                                     /var/log/messages
#*.info;mail.none;authpriv.none;cron.none               /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

Here's the /etc/sysconfig/syslog:

Code:

# Options to syslogd
# -m 0 disables 'MARK' messages.
# -r enables logging from remote machines
# -x disables DNS lookups on messages recieved with -r
# See syslogd(8) for more details
SYSLOGD_OPTIONS="-m 0 -x -r"
# Options to klogd
# -2 prints all kernel oops messages twice; once for klogd to decode, and
#    once for processing with 'ksymoops'
# -x disables all klogd processing of oops messages entirely
# See klogd(8) for more details
#KLOGD_OPTIONS="-x"
KLOGD_OPTIONS="-x"

Any suggestions greatly appreciated.

Thanks,
DT

anomie · 03-21-2009, 11:41 PM

Quote:

Originally Posted by Dark Typhoon

I also made modifications to the /etc/syslog.conf and /etc/sysconfig/syslog files.

Have you tried reverting those changes? Hopefully you've kept the original versions somewhere safe.

Time to start eliminating possibilities to get to the root cause. If reverting the changes and restarting the syslog service does not correct the problem, then it's time to review the packages you most recently updated. They should have been recorded to /var/log/up2date (assuming that logging is working).

Also, what does $ /usr/sbin/getenforce show?

Dark Typhoon · 03-22-2009, 11:45 AM

Quote:

Have you tried reverting those changes? Hopefully you've kept the original versions somewhere safe.

No, I didn't keep the originals. I got sloppy and simply commented out the lines that I was modifying, leaving them there for reference. I have tried removing my changes and uncommenting the original lines, but I do not have bit for bit backups of the originals

Logging to /var/log/up2date ends at the same time logging to everything else does. I believe that this coincides with the time that I instealled Perl. To clarify, I installed Perl 5.8.9 from source as opposed to from up2date. I need a version of Perl that is 5.8.8 or higher for one of the custom packages I am running and I couldn't figure out how to get a version higher than 5.8.5 from up2date.

I also reinitialized VMware Server and made modifications to the compiled Perl path so that VMware server would load 5.8.9 perl modules instead of 5.8.5.

Code:

$ /usr/sbin/getenforce
Enforcing

anomie · 03-22-2009, 03:06 PM

Quote:

Originally Posted by Dark Typhoon

No, I didn't keep the originals. I got sloppy and simply commented out the lines that I was modifying, leaving them there for reference.

Consider using version control for your config files from here going forward. I use rcs (which you can install using up2date), but that's just my preference.

At the very, very least, use some form of sloppy version control prior to editing, e.g.:

# cp -p /etc/foo.conf /etc/foo.conf.20090301 <-- should reflect today's date

You need to be able to roll back changes without thinking too hard about them.

Quote:

Originally Posted by Dark Typhoon

I believe that this coincides with the time that I instealled Perl. To clarify, I installed Perl 5.8.9 from source as opposed to from up2date. I need a version of Perl that is 5.8.8 or higher for one of the custom packages I am running and I couldn't figure out how to get a version higher than 5.8.5 from up2date.

A couple other comments, then: For your sanity, you'll also need to come up with a change management policy. When you change three things all at once and something breaks, it's a lot more work to track down the cause. When you change one thing at a time and perform appropriate regression testing afterward, it is easier to isolate problems.

The problem with diverging from the RHN (with your own custom packages or source installations) is you open a can of worms that makes it significantly more likely you're going to break something. Especially with a crucial app like perl. If you absolutely require a newer version of perl, it may be necessary to start planning an upgrade to RHEL5 (or check your other RHN channels for RHEL4 to see if it happens to be available).

Quote:

Originally Posted by Dark Typhoon

Code:

$ /usr/sbin/getenforce
Enforcing

So you've got selinux on and actively enforcing. Just for grins, try:

# restorecon -R /var/log

Then run the exact same command on the directory you've installed perl to. Finally, restart the syslog service. Problem solved? If not, you could also boot with selinux disabled to try to eliminate that as a possible cause.

Dark Typhoon · 03-22-2009, 05:36 PM

Thanks,

Quote:

Originally Posted by anomie

So you've got selinux on and actively enforcing. Just for grins, try:

# restorecon -R /var/log

Then run the exact same command on the directory you've installed perl to. Finally, restart the syslog service. Problem solved? If not, you could also boot with selinux disabled to try to eliminate that as a possible cause.

Before I enact this change, what precisely will this be doing? The custom packages that I need are currently functioning, I just have to manually start logging after booting. Will this potentially break them?

anomie · 03-22-2009, 08:21 PM

See the manpages for restorecon(8). With that command you'll be recursively restoring default security contexts for the directory specified. It should not break anything unless you've been playing around with selinux policies.

Dark Typhoon · 03-22-2009, 08:25 PM

Oh yeah, man pages... Duh!

Thanks, I will try that out as soon as I get in tomorrow.

Dark Typhoon · 03-23-2009, 01:26 PM

restorecon didn't fix it but disabling SE Linux did. I'll have to figure out what SE Linux settings I messed up with the Perl upgrade but I'm now good to go.

Thanks for you time and patience,
DT

anomie · 03-23-2009, 01:37 PM

One thing you could try is putting selinux in permissive mode. Then watch /var/log/messages for entries of the form "kernel: audit ..."

This should give you clues about what exactly is being denied.

Dark Typhoon · 03-23-2009, 01:41 PM

Cool, thanks, I'll look into as soon as I get a chance.

Cheers,
DT