kern.log remains empty

eentonig · 03-03-2009, 04:22 PM

Since a few days, I noticed that I no longer get any messages in /var/log/kern.log

First I thought it was because I installed syslog-ng, but even reinstalling klogd didn't change anything.

Can anybody point me to the correct steps/locations to troubleshoot this?

(BTW. My Firewall should be logging data, so it's not normal that kern.log is empty.)

unSpawn · 03-03-2009, 05:45 PM

With GNU/Linux reinstalling software to alleviate problems most of the time just is not necessary. Just needs solving "the right way". A simple way to check if the logfile is opened by any process could be to run 'lsof +D/var/log'. Does it show up? Is 'klogd' running? Do log entries show up in other logfiles? Was anything changed in your syslog config with respect to facilities and priorities? If you still can run "old" Syslog with a sane configuration, does it work as advertised?

eentonig · 03-03-2009, 11:48 PM

Thanks Unspawn,

As is obvious, I have no clue what is the right way to continue, so I come here for help. My iptables are logging at level WARNING, and since I run ssh, I regularly have hits on at least those lines, defending me against script kiddies.

I only noticed this behaviour after installing syslog-ng, and after soms troubleshooting, I tried reinstalling klogd and sysklogd to see if the behavior was due to syslog-ng or not.

I thought klogd was responsible for populating kern.log, but syslog-ng insists on removing it. So this seemed like an obvious cause.

Code:

sudo apt-get install syslog-ng    
[sudo] password for rfonteyn: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  klogd sysklogd ubuntu-minimal
The following NEW packages will be installed:
  syslog-ng
0 upgraded, 1 newly installed, 3 to remove and 121 not upgraded.
Need to get 0B/130kB of archives.
After this operation, 16.4kB of additional disk space will be used.

My syslog-ng config is still default to when I installed it.

Code:

rfonteyn@gateway:~$ more /etc/syslog-ng/syslog-ng.conf 
#
# Configuration file for syslog-ng under Debian
#
# attempts at reproducing default syslog behavior

# the standard syslog levels are (in descending order of priority):
# emerg alert crit err warning notice info debug
# the aliases "error", "panic", and "warn" are deprecated
# the "none" priority found in the original syslogd configuration is
# only used in internal messages created by syslogd


######
# options

options {
  time_reopen(10);
  time_reap(360);
  log_fifo_size(2048);
  create_dirs(yes);
  group(adm);
  perm(0640);
  dir_perm(0755);
  use_dns(no);
  stats_freq(0);
  bad_hostname("^gconfd$");
  use_fqdn(yes);
  dns_cache(yes);
  };


######
# sources

# all known message sources
source s_all {
  internal();
  unix-stream("/dev/log");
  file("/var/log/kern.log" log_prefix("kernel: "));
  };


######
# destinations

# some standard log files
destination df_auth { file("/var/log/auth.log"); };
destination df_syslog { file("/var/log/syslog"); };
destination df_cron { file("/var/log/cron.log"); };
destination df_daemon { file("/var/log/daemon.log"); };
destination df_kern { file("/var/log/kern.log"); };
destination df_lpr { file("/var/log/lpr.log"); };
destination df_mail { file("/var/log/mail.log"); };
destination df_user { file("/var/log/user.log"); };
destination df_uucp { file("/var/log/uucp.log"); };

# these files are meant for the mail system log files
# and provide re-usable destinations for {mail,cron,...}.info,
# {mail,cron,...}.notice, etc.
destination df_facility_dot_info { file("/var/log/$FACILITY.info"); };
destination df_facility_dot_notice { file("/var/log/$FACILITY.notice"); };
destination df_facility_dot_warn { file("/var/log/$FACILITY.warn"); };
destination df_facility_dot_err { file("/var/log/$FACILITY.err"); };
destination df_facility_dot_crit { file("/var/log/$FACILITY.crit"); };

# these files are meant for the news system, and are kept separated
# because they should be owned by "news" instead of "root"
destination df_news_dot_notice { file("/var/log/news/news.notice" owner("news")); };
destination df_news_dot_err { file("/var/log/news/news.err" owner("news")); };
destination df_news_dot_crit { file("/var/log/news/news.crit" owner("news")); };

# some more classical and useful files found in standard syslog configurations
destination df_debug { file("/var/log/debug"); };
destination df_messages { file("/var/log/messages"); };

# pipes
# a console to view log messages under X
destination dp_xconsole { pipe("/dev/xconsole"); };

# consoles
# this will send messages to everyone logged in
destination du_all { usertty("*"); };


######
# filters

# all messages from the auth and authpriv facilities
filter f_auth { facility(auth, authpriv); };

# all messages except from the auth and authpriv facilities
filter f_syslog { not facility(auth, authpriv); };

# respectively: messages from the cron, daemon, kern, lpr, mail, news, user,
# and uucp facilities
filter f_cron { facility(cron); };
filter f_daemon { facility(daemon); };
filter f_kern { facility(kern); };
filter f_lpr { facility(lpr); };
filter f_mail { facility(mail); };
filter f_news { facility(news); };
filter f_user { facility(user); };
filter f_uucp { facility(uucp); };

# some filters to select messages of priority greater or equal to info, warn,
# and err
# (equivalents of syslogd's *.info, *.warn, and *.err)
filter f_at_least_info { level(info..emerg); };
filter f_at_least_notice { level(notice..emerg); };
filter f_at_least_warn { level(warn..emerg); };
filter f_at_least_err { level(err..emerg); };
filter f_at_least_crit { level(crit..emerg); };

# all messages of priority debug not coming from the auth, authpriv, news, and
# mail facilities
filter f_debug { level(debug) and not facility(auth, authpriv, news, mail); };

# all messages of info, notice, or warn priority not coming form the auth,
# authpriv, cron, daemon, mail, and news facilities
filter f_messages {
        level(info,notice,warn)
            and not facility(auth,authpriv,cron,daemon,mail,news);
};

# messages with priority emerg
filter f_emerg { level(emerg); };

# complex filter for messages usually sent to the xconsole
filter f_xconsole {
    facility(daemon,mail)
        or level(debug,info,notice,warn)
        or (facility(news)
                and level(crit,err,notice));
};


######
# logs
# order matters if you use "flags(final);" to mark the end of processing in a
# "log" statement

# these rules provide the same behavior as the commented original syslogd rules

# auth,authpriv.*                 /var/log/auth.log
log {
  source(s_all);
  filter(f_auth);
  destination(df_auth);
  };

# *.*;auth,authpriv.none          -/var/log/syslog
log {
  source(s_all);
  filter(f_syslog);
  destination(df_syslog);
  };

# this is commented out in the default syslog.conf
# cron.*                         /var/log/cron.log
#log {
#        source(s_all);
#        filter(f_cron);
#        destination(df_cron);
#};

# daemon.*                        -/var/log/daemon.log
log {
  source(s_all);
  filter(f_daemon);
  destination(df_daemon);
  };

# kern.*                          -/var/log/kern.log
log {
  source(s_all);
  filter(f_kern);
  destination(df_kern);
  };

# lpr.*                           -/var/log/lpr.log
log {
  source(s_all);
  filter(f_lpr);
  destination(df_lpr);
  };

# mail.*                          -/var/log/mail.log
log {
  source(s_all);
  filter(f_mail);
  destination(df_mail);
  };

# user.*                          -/var/log/user.log
log {
  source(s_all);
  filter(f_user);
  destination(df_user);
  };

# uucp.*                          /var/log/uucp.log
log {
  source(s_all);
  filter(f_uucp);
  destination(df_uucp);
  };

# mail.info                       -/var/log/mail.info
log {
  source(s_all);
  filter(f_mail);
  filter(f_at_least_info);
  destination(df_facility_dot_info);
  };

# mail.warn                       -/var/log/mail.warn
log {
  source(s_all);
  filter(f_mail);
  filter(f_at_least_warn);
  destination(df_facility_dot_warn);
  };

# mail.err                        /var/log/mail.err
log {
  source(s_all);
  filter(f_mail);
  filter(f_at_least_err);
  destination(df_facility_dot_err);
  };

# news.crit                       /var/log/news/news.crit
log {
  source(s_all);
  filter(f_news);
  filter(f_at_least_crit);
  destination(df_news_dot_crit);
  };

# news.err                        /var/log/news/news.err
log {
  source(s_all);
  filter(f_news);
  filter(f_at_least_err);
  destination(df_news_dot_err);
  };

# news.notice                     /var/log/news/news.notice
log {
  source(s_all);
  filter(f_news);
  filter(f_at_least_notice);
  destination(df_news_dot_notice);
  };


# *.=debug;\
#         auth,authpriv.none;\
#         news.none;mail.none     -/var/log/debug
log {
  source(s_all);
  filter(f_debug);
  destination(df_debug);
  };


# *.=info;*.=notice;*.=warn;\
#         auth,authpriv.none;\
#         cron,daemon.none;\
#         mail,news.none          -/var/log/messages
log {
  source(s_all);
  filter(f_messages);
  destination(df_messages);
  };

# *.emerg                         *
log {
  source(s_all);
  filter(f_emerg);
  destination(du_all);
  };


# daemon.*;mail.*;\
#         news.crit;news.err;news.notice;\
#         *.=debug;*.=info;\
#         *.=notice;*.=warn       |/dev/xconsole
log {
  source(s_all);
  filter(f_xconsole);
  destination(dp_xconsole);
  };

and for record, the old syslog.conf, which wasn't changed.

Code:

#  /etc/syslog.conf     Configuration file for syslogd.
#
#                       For more information see syslog.conf(5)
#                       manpage.

#
# First some standard logfiles.  Log by facility.
#

auth,authpriv.*                 /var/log/auth.log
*.*;auth,authpriv.none          -/var/log/syslog
#cron.*                         /var/log/cron.log
daemon.*                        -/var/log/daemon.log
kern.*                          -/var/log/kern.log
lpr.*                           -/var/log/lpr.log
mail.*                          -/var/log/mail.log
user.*                          -/var/log/user.log

#
# Logging for the mail system.  Split it up so that
# it is easy to write scripts to parse these files.
#
mail.info                       -/var/log/mail.info
mail.warning                    -/var/log/mail.warn
mail.err                        /var/log/mail.err

# Logging for INN news system
#
news.crit                       /var/log/news/news.crit
news.err                        /var/log/news/news.err
news.notice                     -/var/log/news/news.notice

#
# Some `catch-all' logfiles.
#
*.=debug;\
        auth,authpriv.none;\
        news.none;mail.none     -/var/log/debug
*.=info;*.=notice;*.=warning;\
        auth,authpriv.none;\
        cron,daemon.none;\
        mail,news.none          -/var/log/messages

#
# Emergencies are sent to everybody logged in.
#
*.emerg                         *

#
# I like to have messages displayed on the console, but only on a virtual
# console I usually leave idle.
#
#daemon,mail.*;\
#       news.=crit;news.=err;news.=notice;\
#       *.=debug;*.=info;\
#       *.=notice;*.=warning    /dev/tty8

# The named pipe /dev/xconsole is for the `xconsole' utility.  To use it,
# you must invoke `xconsole' with the `-file' option:
# 
#    $ xconsole -file /dev/xconsole [...]
#
# NOTE: adjust the list below, or you'll go crazy if you have a reasonably
#      busy site..
#
daemon.*;mail.*;\
        news.err;\
        *.=debug;*.=info;\
        *.=notice;*.=warning    |/dev/xconsole

kern.=debug     -/var/log/bandwidth

Code:

root@gateway:~/bin# lsof +D/var/log
COMMAND    PID     USER   FD   TYPE DEVICE    SIZE    NODE NAME
console-k 5153     root   10w   REG    8,1  371687 1974353 /var/log/ConsoleKit/history
squid     5342    proxy    5u   REG    8,1    4581 2015277 /var/log/squid/cache.log
squid     5342    proxy    7w   REG    8,1  838282 2016213 /var/log/squid/access.log
squid     5342    proxy    9w   REG    8,1 1101521 2015294 /var/log/squid/store.log
squid-pre 5366     root    3r   REG    8,1  838282 2016213 /var/log/squid/access.log
apache2   5476     root    2w   REG    8,1     264 2056744 /var/log/apache2/error.log
apache2   5476     root    6w   REG    8,1       0 2056736 /var/log/apache2/other_vhosts_access.log
apache2   5476     root    7w   REG    8,1       0 2056743 /var/log/apache2/access.log
apache2   5478 www-data    2w   REG    8,1     264 2056744 /var/log/apache2/error.log
apache2   5478 www-data    6w   REG    8,1       0 2056736 /var/log/apache2/other_vhosts_access.log
apache2   5478 www-data    7w   REG    8,1       0 2056743 /var/log/apache2/access.log
apache2   5481 www-data    2w   REG    8,1     264 2056744 /var/log/apache2/error.log
apache2   5481 www-data    6w   REG    8,1       0 2056736 /var/log/apache2/other_vhosts_access.log
apache2   5481 www-data    7w   REG    8,1       0 2056743 /var/log/apache2/access.log
apache2   5485 www-data    2w   REG    8,1     264 2056744 /var/log/apache2/error.log
apache2   5485 www-data    6w   REG    8,1       0 2056736 /var/log/apache2/other_vhosts_access.log
apache2   5485 www-data    7w   REG    8,1       0 2056743 /var/log/apache2/access.log
tail      5673 rfonteyn    3r   REG    8,1       0 1975116 /var/log/kern.log
tail      5674 rfonteyn    3r   REG    8,1   26375 1974900 /var/log/syslog
syslog-ng 8539     root    4r   REG    8,1       0 1975116 /var/log/kern.log
syslog-ng 8539     root    5w   REG    8,1   26375 1974900 /var/log/syslog
syslog-ng 8539     root    6w   REG    8,1    7373 1975118 /var/log/messages
syslog-ng 8539     root    9w   REG    8,1   54821 1975112 /var/log/daemon.log
syslog-ng 8539     root   11w   REG    8,1   34904 1974980 /var/log/auth.log

unSpawn · 03-04-2009, 05:10 PM

Pondering the file descriptor being read from, not written to as you would expect with logs:

Code:

COMMAND    PID     USER   FD   TYPE DEVICE    SIZE    NODE NAME
syslog-ng 8539     root    4r   REG    8,1       0 1975116 /var/log/kern.log

and /var/log/kern.log being used as *source* for Syslog-ng:

Code:

source s_all {
  internal();
  unix-stream("/dev/log");
  file("/var/log/kern.log" log_prefix("kernel: "));
  };

I kind of doubt that's right... How about commenting out the kern.log souce line and restarting syslog-ng? See if that works.

eentonig · 03-10-2009, 08:38 AM

Quote:

Originally Posted by unSpawn

Pondering the file descriptor being read from, not written to as you would expect with logs:

Code:

COMMAND    PID     USER   FD   TYPE DEVICE    SIZE    NODE NAME
syslog-ng 8539     root    4r   REG    8,1       0 1975116 /var/log/kern.log

and /var/log/kern.log being used as *source* for Syslog-ng:

Code:

source s_all {
  internal();
  unix-stream("/dev/log");
  file("/var/log/kern.log" log_prefix("kernel: "));
  };

I kind of doubt that's right... How about commenting out the kern.log souce line and restarting syslog-ng? See if that works.

Sorry for the late reply.

Thanks unSpawn. I had already found this myself as well, after reading some lsof manpages. But that was indeed the problem.

I'm left now with the mistery who or what changed this. But so far, can't find any evidence for malicious activity, so most likely the problem was between the keyboard and the chair.