Help me determine why server crashed?

FireRaven · 10-11-2010, 03:21 PM

Hi last night my server went down. Services stopped working, SSH timed out, I plugged monitor and keyboard into and monitor had no signal (like cable wasn't connected) and keyboard didn't light up (not sure if it normally does though).

I rebooted it physically this morning and got these messages from syslog:
#cat /var/log/syslog

Code:

Oct 11 22:17:01 dan CRON[3139]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct 11 23:17:01 dan CRON[3143]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct 12 00:17:01 dan CRON[3148]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct 12 01:17:01 dan CRON[3153]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct 12 02:17:01 dan CRON[3157]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct 12 03:17:01 dan CRON[3162]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct 12 04:17:01 dan CRON[3166]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct 12 07:11:51 dan kernel: imklog 4.2.0, log source = /proc/kmsg started.
Oct 12 07:11:51 dan rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="634" x-info="http://www.rsyslog.com"] (re)start
Oct 12 07:11:51 dan rsyslogd: rsyslogd's groupid changed to 103
Oct 12 07:11:51 dan rsyslogd: rsyslogd's userid changed to 101
Oct 12 07:11:51 dan rsyslogd-2039: Could no open output file '/dev/xconsole' [try http://www.rsyslog.com/e/2039 ]
Oct 12 07:11:51 dan kernel: [    0.000000] Initializing cgroup subsys cpuset
Oct 12 07:11:51 dan kernel: [    0.000000] Initializing cgroup subsys cpu

I shortened the above messages, but you can see it went down around 4:17am where the last cron ran. And at 7:11 was when I rebooted.

Is there any more info I can get that would give me more information?

Can I determine whether it was a Kernel Panic?

rn_ · 10-12-2010, 09:42 AM

what does /var/log/messages say around that time?

quanta · 10-12-2010, 10:48 AM

Quote:

Originally Posted by rn_

what does /var/log/messages say around that time?

And have a look at /var/log/dmesg also.

FireRaven · 10-12-2010, 05:02 PM

/vat/log/messages shows:

Code:

Oct 10 06:25:48 dan rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="627" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.
Oct 11 06:48:54 dan rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="627" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.
Oct 12 07:11:51 dan kernel: imklog 4.2.0, log source = /proc/kmsg started.

As you can see it only has one line logged for the 11th when it happened.

Quote:

Originally Posted by quanta

And have a look at /var/log/dmesg also.

Here's the dmesg but I can't see it showing anything either:

Code:

[   15.712108] r8169: eth0: link up
[   15.712131] r8169: eth0: link up
[   15.773969] No connectors reported connected with modes
[   15.774036] [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on minor 0
[   15.774525] HDA Intel 0000:00:1b.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[   15.774592] HDA Intel 0000:00:1b.0: setting latency timer to 64
[   15.780332] vga16fb: initializing
[   15.780351] vga16fb: mapped to 0xffff8800000a0000
[   15.780556] fb0: VGA16 VGA frame buffer device
[   15.794708] Slow work thread pool: Starting up
[   15.795395] Slow work thread pool: Ready
[   15.918442] type=1505 audit(1286181761.274:5):  operation="profile_replace" pid=683 name="/sbin/dhclient3"
[   15.919062] type=1505 audit(1286181761.274:6):  operation="profile_replace" pid=683 name="/usr/lib/NetworkManager/nm-dhcp-client.action"
[   15.919420] type=1505 audit(1286181761.274:7):  operation="profile_replace" pid=683 name="/usr/lib/connman/scripts/dhclient-script"
[   15.923594] type=1505 audit(1286181761.284:8):  operation="profile_load" pid=684 name="/usr/sbin/tcpdump"
[   15.927546] Console: switching to colour frame buffer device 80x30
[   15.967769] input: HDA Digital PCBeep as /devices/pci0000:00/0000:00:1b.0/input/input4

quanta · 10-12-2010, 09:00 PM

Which distro are you using?

It seems that you haven't configured to log anything to /var/log/messages. Post your syslog configuration file (/etc/(r)syslog.conf).

FireRaven · 10-12-2010, 10:42 PM

Quote:

Originally Posted by quanta

Which distro are you using?

It seems that you haven't configured to log anything to /var/log/messages. Post your syslog configuration file (/etc/(r)syslog.conf).

I'm using Ubuntu Server 10.04 AMD64.

Here are the content of my /etc/rsyslog.conf:

Code:

$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
#$ModLoad immark  # provides --MARK-- message capability

$KLogPath /proc/kmsg

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Filter duplicated messages
$RepeatedMsgReduction on

$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog

$IncludeConfig /etc/rsyslog.d/*.conf

And this is what is in /etc/rsyslog.d/50-default.conf:

Code:

auth,authpriv.*                 /var/log/auth.log
*.*;auth,authpriv.none          -/var/log/syslog
#cron.*                         /var/log/cron.log
daemon.*                        -/var/log/daemon.log
kern.*                          -/var/log/kern.log
lpr.*                           -/var/log/lpr.log
mail.*                          -/var/log/mail.log
user.*                          -/var/log/user.log

mail.info                       -/var/log/mail.info
mail.warn                       -/var/log/mail.warn
mail.err                        /var/log/mail.err

news.crit                       /var/log/news/news.crit
news.err                        /var/log/news/news.err
news.notice                     -/var/log/news/news.notice

*.=debug;\
        auth,authpriv.none;\
        news.none;mail.none     -/var/log/debug
*.=info;*.=notice;*.=warn;\
        auth,authpriv.none;\
        cron,daemon.none;\
        mail,news.none          -/var/log/messages

*.emerg                         *

daemon.*;mail.*;\
        news.err;\
        *.=debug;*.=info;\
        *.=notice;*.=warn       |/dev/xconsole

And in /etc/rsyslog.d/20-ufw.conf:

Code:

# Log kernel generated UFW log messages to file
:msg,contains,"[UFW " /var/log/ufw.log

# Uncomment the following to stop logging anything that matches the last rule.
# Doing this will stop logging kernel generated UFW log messages to the file
# normally containing kern.* messages (eg, /var/log/kern.log)
#& ~

There is also (but probably not important) /etc/rsyslog.d/postfix:

Code:

# Create an additional socket in postfix's chroot in order not to break
# mail logging when rsyslog is restarted.  If the directory is missing,
# rsyslog will silently skip creating the socket.
$AddUnixListenSocket /var/spool/postfix/dev/log

FireRaven · 10-12-2010, 10:46 PM

Also if it helps /var/log/ufw.log is empty.
And /var/log/kern.log doesn't have anything other than a stripped down version of /var/log/syslog. Which is showing nothing at time of crash, only the bootup.

quanta · 10-13-2010, 10:50 AM

FYI: https://bugs.launchpad.net/ubuntu/+s...og/+bug/407862

fsargent · 11-12-2010, 03:07 PM

:msg,contains,"[UFW " /var/log/ufw.log
^this line breaks things for me in rsyslog. It'll cause a syntax error when generating the rules in rsyslog. This can be viewed by running it in interactive mode.

Is there any other way I can tell UFW to output to another file, or more to the point, to a remote server (which is my ultimate goal)?