/var/log/messages not updating in CentOS

george_john · 03-01-2018, 12:38 AM

Hi, I've a busy CentOS 5.9 server which is hosting an IoT application server and Oracle database. The /var/log/messages is not logging any normal messages except some logs from Nagios nrpe plugin (This server being monitored by Nagios core).

nrpe[6565]: Error: Request packet type/version was invalid!
nrpe[6565]: Client request was invalid, bailing out...

when i restart any service (eg: crond, cups, sendmail, etc..), it is NOT captured in /var/log/messages.

But when i use logger command, it is captured,

# logger -i "hello"
/var/log/messages >> Mar 1 01:26:35 server01 root[7638]: hello

Here are my rsyslog settings,

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages

# lsof /var/log/messages
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
rsyslogd 30429 root 3w REG 253,0 193500 5304282 /var/log/messages

Any idea what is gone wrong here?

MensaWater · 03-01-2018, 08:24 AM

The default logger for RHEL5/CentOS5 was syslogd.

You indicate you have rsyslog running.

We have one RHEL6 server running rsyslog by default but we also have it running syslog-ng to capture information from network devices. On that system /var/log/messages is open by rsyslog whereas on RHEL5/CentOS5 it was open by syslogd.

On that system the order we start the two different log daemons is important to get things locally logged different than those logged by syslog-ng.

Is it possible you're running both syslogd and rsyslog? Or any other logger? If so order may be important if you need both. Also it may just be you need to stop one and used chkconfig to prevent it from restarting after boot.

FYI: cron start is logged in /var/log/cron not messages. Similarly most mail activity is in /var/log/maillog. You may just be looking for info in the wrong logs.

P.S. CentOS5 is a binary compile from RHEL5 source. RHEL5 went EOL more than a year ago. There are no security or bug fixes for RHEL5 so there are none for CentOS5. You really ought to go to at least CentOS6 or better yet CentOS7 if possible.

george_john · 03-01-2018, 11:15 PM

Thanks for your reply.

It was syslogd running but for a testing I’ve installed rsyslog. Issue persists and I rolled it back to syslogd.

# lsof /var/log/messages
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
syslogd 30269 root 1w REG 253,0 218121 5304282 /var/log/messages

I found that it is not possible to start rsyslog while syslog is running. So I stopped syslog and started rsyslog.

Answer to cron and sendmail logging --> When I do a restart of these services, there are some logs writing to /var/log/messages (checked in another server). But these logs are not captured in this server (/var/log/messages or /var/log/cron).

Ex: # service crond restart
Redirecting to /bin/systemctl restart crond.service

# tail /var/log/messages
Mar 1 23:46:58 system02-DB2 systemd: Stopping Command Scheduler...
Mar 1 23:46:58 system02-DB2 systemd: Started Command Scheduler.
Mar 1 23:46:58 system02-DB2 systemd: Starting Command Scheduler...

Cron and sendmail core activities are captured in their respective log files.

I’m planning to upgrade the OS to CentOS 6 or 7. If it is fixed after upgrade, I’ll mark this post/thread as Resolved.

Thanks.

syg00 · 03-01-2018, 11:26 PM

Quote:

Originally Posted by george_john

Ex: # service crond restart
Redirecting to /bin/systemctl restart crond.service

Say what ?. You mention 5.9 above - have you manually installed systemd ?.

Try journalctl for your logs, although loggers should be honoured regardless if properly configured.

george_john · 03-02-2018, 12:40 AM

Sorry, those commands are from another server (RH 7.3). I just wanted to show that other servers are capturing the logs properly.

RH 7.3 rsyslog.conf,

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages

We can see the configuration is same for CentOS 5.9.

MensaWater · 03-03-2018, 08:49 AM

First RH 7.3 = ancient RedHat 7.3 version. Believe it or not some folks may still run that. RHEL 7.3 = more recent RedHat Enterprise Linux (RHEL7 being the latest RHEL and 7.3 being a subversion of that).

The way rsyslog gets setup on RHEL5/CentOS5 and the version would be different from that used on RHEL6/CentOS6 which in turn would be different than that on RHEL7/CentOS7.
On an old CentOS5 workstation rpm shows rsyslog version: rsyslog-3.22.1-7.el5
On the RHEL6 server rpm shows rsyslog version: rsyslog-5.8.10-8.el6.x86_64
On a RHEL7 workstation rpm shows rsyslog version: rsyslog-7.4.7-12.el7.x86_64

That shows the upstream version of rsyslog is signifcantly different on the 3 major RHEL/CentOS releases.

On the CentOS5 box the default /etc/rsyslog.conf file contains:

Code:

# Use traditional timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Provides kernel logging support (previously done by rklogd)
$ModLoad imklog
# Provides support for local system logging (e.g. via logger command)
$ModLoad imuxsock

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

On the other hand /etc/syslog.conf from that same system contains:

Code:

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

Also on RHEL5/CentOS5 /etc/sysconfig/rsyslog (and /etc/sysconfig/syslog) are used. (On RHEL6/CentOS6 the file exists but says OPTIONS are deprecated so doesn't really set any). The one from my CentOS5 contains:

Code:

# Options to syslogd
# -m 0 disables 'MARK' messages.
# -rPortNumber Enables logging from remote machines. The listener will listen to the specified port.
# -x disables DNS lookups on messages recieved with -r
# See syslogd(8) for more details
SYSLOGD_OPTIONS="-m 0"
# Options to klogd
# -2 prints all kernel oops messages twice; once for klogd to decode, and
#    once for processing with 'ksymoops'
# -x disables all klogd processing of oops messages entirely
# See klogd(8) for more details
KLOGD_OPTIONS="-x"

I'm wondering if you're modifying /etc/syslog.conf rather than /etc/rsyslog.conf. Although the files are similar the init scripts on start show they use the file with same name as the logger. You may want to use the above files from CentOS5 for rsyslog to verify yours are correct. Note that I don't actually have rsyslog turned on (via chkconfig) on my CentOS5. I just installed the package to see what it gives. On that system I use the default syslog rather than rsyslog.

george_john · 03-05-2018, 04:32 AM

I've uninstalled rsyslog from my problematic server (CentOS 5.9). Now it is running with the default one (syslog). Issue still not resolved.

# service syslog status
syslogd (pid 30269) is running...
klogd (pid 30272) is running...

# cat /etc/syslog.conf
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages

# The authpriv file has restricted access.
authpriv.* /var/log/secure

# Log all the mail messages in one place.
mail.* -/var/log/maillog

# Log cron stuff
cron.* /var/log/cron

# Everybody gets emergency messages
*.emerg *

# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler

# Save boot messages also to boot.log
local7.* /var/log/boot.log

MensaWater · 03-05-2018, 07:48 AM

Is what you just posted /etc/syslog.conf? /etc/rsyslog.conf? Something else?

Did you verify rsyslog has been stopped and syslog RE-started?

Did you run chkconfig to verify syslog is set to start on reboot and rsyslog isn't?

What you posted seems to be missing quite a bit from the default syslog.conf I posted. I suggest you do a line by line comparison and add missing lines to your file at a minimum.