LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Software (https://www.linuxquestions.org/questions/linux-software-2/)
-   -   kern.log remains empty - where to start troubleshooting? (https://www.linuxquestions.org/questions/linux-software-2/kern-log-remains-empty-where-to-start-troubleshooting-708918/)

eentonig 03-03-2009 04:22 PM
[SOLVED] kern.log remains empty - where to start troubleshooting?
 
Since a few days, I noticed that I no longer get any messages in /var/log/kern.log

First I thought it was because I installed syslog-ng, but even reinstalling klogd didn't change anything.

Can anybody point me to the correct steps/locations to troubleshoot this?

(BTW. My Firewall should be logging data, so it's not normal that kern.log is empty.)

unSpawn 03-03-2009 05:45 PM
With GNU/Linux reinstalling software to alleviate problems most of the time just is not necessary. Just needs solving "the right way". A simple way to check if the logfile is opened by any process could be to run 'lsof +D/var/log'. Does it show up? Is 'klogd' running? Do log entries show up in other logfiles? Was anything changed in your syslog config with respect to facilities and priorities? If you still can run "old" Syslog with a sane configuration, does it work as advertised?

eentonig 03-03-2009 11:48 PM
Thanks Unspawn,

As is obvious, I have no clue what is the right way to continue, so I come here for help. My iptables are logging at level WARNING, and since I run ssh, I regularly have hits on at least those lines, defending me against script kiddies.

I only noticed this behaviour after installing syslog-ng, and after soms troubleshooting, I tried reinstalling klogd and sysklogd to see if the behavior was due to syslog-ng or not.

I thought klogd was responsible for populating kern.log, but syslog-ng insists on removing it. So this seemed like an obvious cause.


Code:
sudo apt-get install syslog-ng   
[sudo] password for rfonteyn:
Reading package lists... Done
Building dependency tree     
Reading state information... Done
The following packages will be REMOVED:
  klogd sysklogd ubuntu-minimal
The following NEW packages will be installed:
  syslog-ng
0 upgraded, 1 newly installed, 3 to remove and 121 not upgraded.
Need to get 0B/130kB of archives.
After this operation, 16.4kB of additional disk space will be used.
My syslog-ng config is still default to when I installed it.
Code:
rfonteyn@gateway:~$ more /etc/syslog-ng/syslog-ng.conf
#
# Configuration file for syslog-ng under Debian
#
# attempts at reproducing default syslog behavior

# the standard syslog levels are (in descending order of priority):
# emerg alert crit err warning notice info debug
# the aliases "error", "panic", and "warn" are deprecated
# the "none" priority found in the original syslogd configuration is
# only used in internal messages created by syslogd


######
# options

options {
  time_reopen(10);
  time_reap(360);
  log_fifo_size(2048);
  create_dirs(yes);
  group(adm);
  perm(0640);
  dir_perm(0755);
  use_dns(no);
  stats_freq(0);
  bad_hostname("^gconfd$");
  use_fqdn(yes);
  dns_cache(yes);
  };


######
# sources

# all known message sources
source s_all {
  internal();
  unix-stream("/dev/log");
  file("/var/log/kern.log" log_prefix("kernel: "));
  };


######
# destinations

# some standard log files
destination df_auth { file("/var/log/auth.log"); };
destination df_syslog { file("/var/log/syslog"); };
destination df_cron { file("/var/log/cron.log"); };
destination df_daemon { file("/var/log/daemon.log"); };
destination df_kern { file("/var/log/kern.log"); };
destination df_lpr { file("/var/log/lpr.log"); };
destination df_mail { file("/var/log/mail.log"); };
destination df_user { file("/var/log/user.log"); };
destination df_uucp { file("/var/log/uucp.log"); };

# these files are meant for the mail system log files
# and provide re-usable destinations for {mail,cron,...}.info,
# {mail,cron,...}.notice, etc.
destination df_facility_dot_info { file("/var/log/$FACILITY.info"); };
destination df_facility_dot_notice { file("/var/log/$FACILITY.notice"); };
destination df_facility_dot_warn { file("/var/log/$FACILITY.warn"); };
destination df_facility_dot_err { file("/var/log/$FACILITY.err"); };
destination df_facility_dot_crit { file("/var/log/$FACILITY.crit"); };

# these files are meant for the news system, and are kept separated
# because they should be owned by "news" instead of "root"
destination df_news_dot_notice { file("/var/log/news/news.notice" owner("news")); };
destination df_news_dot_err { file("/var/log/news/news.err" owner("news")); };
destination df_news_dot_crit { file("/var/log/news/news.crit" owner("news")); };

# some more classical and useful files found in standard syslog configurations
destination df_debug { file("/var/log/debug"); };
destination df_messages { file("/var/log/messages"); };

# pipes
# a console to view log messages under X
destination dp_xconsole { pipe("/dev/xconsole"); };

# consoles
# this will send messages to everyone logged in
destination du_all { usertty("*"); };


######
# filters

# all messages from the auth and authpriv facilities
filter f_auth { facility(auth, authpriv); };

# all messages except from the auth and authpriv facilities
filter f_syslog { not facility(auth, authpriv); };

# respectively: messages from the cron, daemon, kern, lpr, mail, news, user,
# and uucp facilities
filter f_cron { facility(cron); };
filter f_daemon { facility(daemon); };
filter f_kern { facility(kern); };
filter f_lpr { facility(lpr); };
filter f_mail { facility(mail); };
filter f_news { facility(news); };
filter f_user { facility(user); };
filter f_uucp { facility(uucp); };

# some filters to select messages of priority greater or equal to info, warn,
# and err
# (equivalents of syslogd's *.info, *.warn, and *.err)
filter f_at_least_info { level(info..emerg); };
filter f_at_least_notice { level(notice..emerg); };
filter f_at_least_warn { level(warn..emerg); };
filter f_at_least_err { level(err..emerg); };
filter f_at_least_crit { level(crit..emerg); };

# all messages of priority debug not coming from the auth, authpriv, news, and
# mail facilities
filter f_debug { level(debug) and not facility(auth, authpriv, news, mail); };

# all messages of info, notice, or warn priority not coming form the auth,
# authpriv, cron, daemon, mail, and news facilities
filter f_messages {
        level(info,notice,warn)
            and not facility(auth,authpriv,cron,daemon,mail,news);
};

# messages with priority emerg
filter f_emerg { level(emerg); };

# complex filter for messages usually sent to the xconsole
filter f_xconsole {
    facility(daemon,mail)
        or level(debug,info,notice,warn)
        or (facility(news)
                and level(crit,err,notice));
};


######
# logs
# order matters if you use "flags(final);" to mark the end of processing in a
# "log" statement

# these rules provide the same behavior as the commented original syslogd rules

# auth,authpriv.*                /var/log/auth.log
log {
  source(s_all);
  filter(f_auth);
  destination(df_auth);
  };

# *.*;auth,authpriv.none          -/var/log/syslog
log {
  source(s_all);
  filter(f_syslog);
  destination(df_syslog);
  };

# this is commented out in the default syslog.conf
# cron.*                        /var/log/cron.log
#log {
#        source(s_all);
#        filter(f_cron);
#        destination(df_cron);
#};

# daemon.*                        -/var/log/daemon.log
log {
  source(s_all);
  filter(f_daemon);
  destination(df_daemon);
  };

# kern.*                          -/var/log/kern.log
log {
  source(s_all);
  filter(f_kern);
  destination(df_kern);
  };

# lpr.*                          -/var/log/lpr.log
log {
  source(s_all);
  filter(f_lpr);
  destination(df_lpr);
  };

# mail.*                          -/var/log/mail.log
log {
  source(s_all);
  filter(f_mail);
  destination(df_mail);
  };

# user.*                          -/var/log/user.log
log {
  source(s_all);
  filter(f_user);
  destination(df_user);
  };

# uucp.*                          /var/log/uucp.log
log {
  source(s_all);
  filter(f_uucp);
  destination(df_uucp);
  };

# mail.info                      -/var/log/mail.info
log {
  source(s_all);
  filter(f_mail);
  filter(f_at_least_info);
  destination(df_facility_dot_info);
  };

# mail.warn                      -/var/log/mail.warn
log {
  source(s_all);
  filter(f_mail);
  filter(f_at_least_warn);
  destination(df_facility_dot_warn);
  };

# mail.err                        /var/log/mail.err
log {
  source(s_all);
  filter(f_mail);
  filter(f_at_least_err);
  destination(df_facility_dot_err);
  };

# news.crit                      /var/log/news/news.crit
log {
  source(s_all);
  filter(f_news);
  filter(f_at_least_crit);
  destination(df_news_dot_crit);
  };

# news.err                        /var/log/news/news.err
log {
  source(s_all);
  filter(f_news);
  filter(f_at_least_err);
  destination(df_news_dot_err);
  };

# news.notice                    /var/log/news/news.notice
log {
  source(s_all);
  filter(f_news);
  filter(f_at_least_notice);
  destination(df_news_dot_notice);
  };


# *.=debug;\
#        auth,authpriv.none;\
#        news.none;mail.none    -/var/log/debug
log {
  source(s_all);
  filter(f_debug);
  destination(df_debug);
  };


# *.=info;*.=notice;*.=warn;\
#        auth,authpriv.none;\
#        cron,daemon.none;\
#        mail,news.none          -/var/log/messages
log {
  source(s_all);
  filter(f_messages);
  destination(df_messages);
  };

# *.emerg                        *
log {
  source(s_all);
  filter(f_emerg);
  destination(du_all);
  };


# daemon.*;mail.*;\
#        news.crit;news.err;news.notice;\
#        *.=debug;*.=info;\
#        *.=notice;*.=warn      |/dev/xconsole
log {
  source(s_all);
  filter(f_xconsole);
  destination(dp_xconsole);
  };
and for record, the old syslog.conf, which wasn't changed.
Code:
#  /etc/syslog.conf    Configuration file for syslogd.
#
#                      For more information see syslog.conf(5)
#                      manpage.

#
# First some standard logfiles.  Log by facility.
#

auth,authpriv.*                /var/log/auth.log
*.*;auth,authpriv.none          -/var/log/syslog
#cron.*                        /var/log/cron.log
daemon.*                        -/var/log/daemon.log
kern.*                          -/var/log/kern.log
lpr.*                          -/var/log/lpr.log
mail.*                          -/var/log/mail.log
user.*                          -/var/log/user.log

#
# Logging for the mail system.  Split it up so that
# it is easy to write scripts to parse these files.
#
mail.info                      -/var/log/mail.info
mail.warning                    -/var/log/mail.warn
mail.err                        /var/log/mail.err

# Logging for INN news system
#
news.crit                      /var/log/news/news.crit
news.err                        /var/log/news/news.err
news.notice                    -/var/log/news/news.notice

#
# Some `catch-all' logfiles.
#
*.=debug;\
        auth,authpriv.none;\
        news.none;mail.none    -/var/log/debug
*.=info;*.=notice;*.=warning;\
        auth,authpriv.none;\
        cron,daemon.none;\
        mail,news.none          -/var/log/messages

#
# Emergencies are sent to everybody logged in.
#
*.emerg                        *

#
# I like to have messages displayed on the console, but only on a virtual
# console I usually leave idle.
#
#daemon,mail.*;\
#      news.=crit;news.=err;news.=notice;\
#      *.=debug;*.=info;\
#      *.=notice;*.=warning    /dev/tty8

# The named pipe /dev/xconsole is for the `xconsole' utility.  To use it,
# you must invoke `xconsole' with the `-file' option:
#
#    $ xconsole -file /dev/xconsole [...]
#
# NOTE: adjust the list below, or you'll go crazy if you have a reasonably
#      busy site..
#
daemon.*;mail.*;\
        news.err;\
        *.=debug;*.=info;\
        *.=notice;*.=warning    |/dev/xconsole

kern.=debug    -/var/log/bandwidth
Code:
root@gateway:~/bin# lsof +D/var/log
COMMAND    PID    USER  FD  TYPE DEVICE    SIZE    NODE NAME
console-k 5153    root  10w  REG    8,1  371687 1974353 /var/log/ConsoleKit/history
squid    5342    proxy    5u  REG    8,1    4581 2015277 /var/log/squid/cache.log
squid    5342    proxy    7w  REG    8,1  838282 2016213 /var/log/squid/access.log
squid    5342    proxy    9w  REG    8,1 1101521 2015294 /var/log/squid/store.log
squid-pre 5366    root    3r  REG    8,1  838282 2016213 /var/log/squid/access.log
apache2  5476    root    2w  REG    8,1    264 2056744 /var/log/apache2/error.log
apache2  5476    root    6w  REG    8,1      0 2056736 /var/log/apache2/other_vhosts_access.log
apache2  5476    root    7w  REG    8,1      0 2056743 /var/log/apache2/access.log
apache2  5478 www-data    2w  REG    8,1    264 2056744 /var/log/apache2/error.log
apache2  5478 www-data    6w  REG    8,1      0 2056736 /var/log/apache2/other_vhosts_access.log
apache2  5478 www-data    7w  REG    8,1      0 2056743 /var/log/apache2/access.log
apache2  5481 www-data    2w  REG    8,1    264 2056744 /var/log/apache2/error.log
apache2  5481 www-data    6w  REG    8,1      0 2056736 /var/log/apache2/other_vhosts_access.log
apache2  5481 www-data    7w  REG    8,1      0 2056743 /var/log/apache2/access.log
apache2  5485 www-data    2w  REG    8,1    264 2056744 /var/log/apache2/error.log
apache2  5485 www-data    6w  REG    8,1      0 2056736 /var/log/apache2/other_vhosts_access.log
apache2  5485 www-data    7w  REG    8,1      0 2056743 /var/log/apache2/access.log
tail      5673 rfonteyn    3r  REG    8,1      0 1975116 /var/log/kern.log
tail      5674 rfonteyn    3r  REG    8,1  26375 1974900 /var/log/syslog
syslog-ng 8539    root    4r  REG    8,1      0 1975116 /var/log/kern.log
syslog-ng 8539    root    5w  REG    8,1  26375 1974900 /var/log/syslog
syslog-ng 8539    root    6w  REG    8,1    7373 1975118 /var/log/messages
syslog-ng 8539    root    9w  REG    8,1  54821 1975112 /var/log/daemon.log
syslog-ng 8539    root  11w  REG    8,1  34904 1974980 /var/log/auth.log

unSpawn 03-04-2009 05:10 PM
Pondering the file descriptor being read from, not written to as you would expect with logs:
Code:
COMMAND    PID    USER  FD  TYPE DEVICE    SIZE    NODE NAME
syslog-ng 8539    root    4r  REG    8,1      0 1975116 /var/log/kern.log
and /var/log/kern.log being used as *source* for Syslog-ng:
Code:
source s_all {
  internal();
  unix-stream("/dev/log");
  file("/var/log/kern.log" log_prefix("kernel: "));
  };
I kind of doubt that's right... How about commenting out the kern.log souce line and restarting syslog-ng? See if that works.

eentonig 03-10-2009 08:38 AM
Quote:
Originally Posted by unSpawn (Post 3465115)
Pondering the file descriptor being read from, not written to as you would expect with logs:
Code:
COMMAND    PID    USER  FD  TYPE DEVICE    SIZE    NODE NAME
syslog-ng 8539    root    4r  REG    8,1      0 1975116 /var/log/kern.log
and /var/log/kern.log being used as *source* for Syslog-ng:
Code:
source s_all {
  internal();
  unix-stream("/dev/log");
  file("/var/log/kern.log" log_prefix("kernel: "));
  };
I kind of doubt that's right... How about commenting out the kern.log souce line and restarting syslog-ng? See if that works.
Sorry for the late reply.

Thanks unSpawn. I had already found this myself as well, after reading some lsof manpages. But that was indeed the problem.

I'm left now with the mistery who or what changed this. But so far, can't find any evidence for malicious activity, so most likely the problem was between the keyboard and the chair.


All times are GMT -5. The time now is 06:16 PM.