LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 10-11-2010, 04:21 PM   #1
FireRaven
Member
 
Registered: Apr 2006
Location: Australia
Distribution: Debian Squeeze
Posts: 135

Rep: Reputation: 18
Help me determine why server crashed?


Hi last night my server went down. Services stopped working, SSH timed out, I plugged monitor and keyboard into and monitor had no signal (like cable wasn't connected) and keyboard didn't light up (not sure if it normally does though).

I rebooted it physically this morning and got these messages from syslog:
#cat /var/log/syslog
Code:
Oct 11 22:17:01 dan CRON[3139]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct 11 23:17:01 dan CRON[3143]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct 12 00:17:01 dan CRON[3148]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct 12 01:17:01 dan CRON[3153]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct 12 02:17:01 dan CRON[3157]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct 12 03:17:01 dan CRON[3162]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct 12 04:17:01 dan CRON[3166]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct 12 07:11:51 dan kernel: imklog 4.2.0, log source = /proc/kmsg started.
Oct 12 07:11:51 dan rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="634" x-info="http://www.rsyslog.com"] (re)start
Oct 12 07:11:51 dan rsyslogd: rsyslogd's groupid changed to 103
Oct 12 07:11:51 dan rsyslogd: rsyslogd's userid changed to 101
Oct 12 07:11:51 dan rsyslogd-2039: Could no open output file '/dev/xconsole' [try http://www.rsyslog.com/e/2039 ]
Oct 12 07:11:51 dan kernel: [    0.000000] Initializing cgroup subsys cpuset
Oct 12 07:11:51 dan kernel: [    0.000000] Initializing cgroup subsys cpu
I shortened the above messages, but you can see it went down around 4:17am where the last cron ran. And at 7:11 was when I rebooted.

Is there any more info I can get that would give me more information?

Can I determine whether it was a Kernel Panic?
 
Old 10-12-2010, 10:42 AM   #2
rn_
Member
 
Registered: Jun 2009
Location: Orlando, FL, USA
Distribution: Suse, Redhat
Posts: 127
Blog Entries: 1

Rep: Reputation: 25
what does /var/log/messages say around that time?
 
Old 10-12-2010, 11:48 AM   #3
quanta
Member
 
Registered: Aug 2007
Location: Vietnam
Distribution: RedHat based, Debian based, Slackware, Gentoo
Posts: 724

Rep: Reputation: 100Reputation: 100
Quote:
Originally Posted by rn_ View Post
what does /var/log/messages say around that time?
And have a look at /var/log/dmesg also.
 
Old 10-12-2010, 06:02 PM   #4
FireRaven
Member
 
Registered: Apr 2006
Location: Australia
Distribution: Debian Squeeze
Posts: 135

Original Poster
Rep: Reputation: 18
/vat/log/messages shows:
Code:
Oct 10 06:25:48 dan rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="627" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.
Oct 11 06:48:54 dan rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="627" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.
Oct 12 07:11:51 dan kernel: imklog 4.2.0, log source = /proc/kmsg started.
As you can see it only has one line logged for the 11th when it happened.


Quote:
Originally Posted by quanta View Post
And have a look at /var/log/dmesg also.
Here's the dmesg but I can't see it showing anything either:
Code:
[   15.712108] r8169: eth0: link up
[   15.712131] r8169: eth0: link up
[   15.773969] No connectors reported connected with modes
[   15.774036] [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on minor 0
[   15.774525] HDA Intel 0000:00:1b.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[   15.774592] HDA Intel 0000:00:1b.0: setting latency timer to 64
[   15.780332] vga16fb: initializing
[   15.780351] vga16fb: mapped to 0xffff8800000a0000
[   15.780556] fb0: VGA16 VGA frame buffer device
[   15.794708] Slow work thread pool: Starting up
[   15.795395] Slow work thread pool: Ready
[   15.918442] type=1505 audit(1286181761.274:5):  operation="profile_replace" pid=683 name="/sbin/dhclient3"
[   15.919062] type=1505 audit(1286181761.274:6):  operation="profile_replace" pid=683 name="/usr/lib/NetworkManager/nm-dhcp-client.action"
[   15.919420] type=1505 audit(1286181761.274:7):  operation="profile_replace" pid=683 name="/usr/lib/connman/scripts/dhclient-script"
[   15.923594] type=1505 audit(1286181761.284:8):  operation="profile_load" pid=684 name="/usr/sbin/tcpdump"
[   15.927546] Console: switching to colour frame buffer device 80x30
[   15.967769] input: HDA Digital PCBeep as /devices/pci0000:00/0000:00:1b.0/input/input4
 
Old 10-12-2010, 10:00 PM   #5
quanta
Member
 
Registered: Aug 2007
Location: Vietnam
Distribution: RedHat based, Debian based, Slackware, Gentoo
Posts: 724

Rep: Reputation: 100Reputation: 100
Which distro are you using?

It seems that you haven't configured to log anything to /var/log/messages. Post your syslog configuration file (/etc/(r)syslog.conf).
 
Old 10-12-2010, 11:42 PM   #6
FireRaven
Member
 
Registered: Apr 2006
Location: Australia
Distribution: Debian Squeeze
Posts: 135

Original Poster
Rep: Reputation: 18
Quote:
Originally Posted by quanta View Post
Which distro are you using?

It seems that you haven't configured to log anything to /var/log/messages. Post your syslog configuration file (/etc/(r)syslog.conf).
I'm using Ubuntu Server 10.04 AMD64.

Here are the content of my /etc/rsyslog.conf:
Code:
$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
#$ModLoad immark  # provides --MARK-- message capability

$KLogPath /proc/kmsg

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Filter duplicated messages
$RepeatedMsgReduction on

$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog

$IncludeConfig /etc/rsyslog.d/*.conf

And this is what is in /etc/rsyslog.d/50-default.conf:
Code:
auth,authpriv.*                 /var/log/auth.log
*.*;auth,authpriv.none          -/var/log/syslog
#cron.*                         /var/log/cron.log
daemon.*                        -/var/log/daemon.log
kern.*                          -/var/log/kern.log
lpr.*                           -/var/log/lpr.log
mail.*                          -/var/log/mail.log
user.*                          -/var/log/user.log

mail.info                       -/var/log/mail.info
mail.warn                       -/var/log/mail.warn
mail.err                        /var/log/mail.err

news.crit                       /var/log/news/news.crit
news.err                        /var/log/news/news.err
news.notice                     -/var/log/news/news.notice

*.=debug;\
        auth,authpriv.none;\
        news.none;mail.none     -/var/log/debug
*.=info;*.=notice;*.=warn;\
        auth,authpriv.none;\
        cron,daemon.none;\
        mail,news.none          -/var/log/messages

*.emerg                         *

daemon.*;mail.*;\
        news.err;\
        *.=debug;*.=info;\
        *.=notice;*.=warn       |/dev/xconsole
And in /etc/rsyslog.d/20-ufw.conf:
Code:
# Log kernel generated UFW log messages to file
:msg,contains,"[UFW " /var/log/ufw.log

# Uncomment the following to stop logging anything that matches the last rule.
# Doing this will stop logging kernel generated UFW log messages to the file
# normally containing kern.* messages (eg, /var/log/kern.log)
#& ~
There is also (but probably not important) /etc/rsyslog.d/postfix:
Code:
# Create an additional socket in postfix's chroot in order not to break
# mail logging when rsyslog is restarted.  If the directory is missing,
# rsyslog will silently skip creating the socket.
$AddUnixListenSocket /var/spool/postfix/dev/log
 
Old 10-12-2010, 11:46 PM   #7
FireRaven
Member
 
Registered: Apr 2006
Location: Australia
Distribution: Debian Squeeze
Posts: 135

Original Poster
Rep: Reputation: 18
Also if it helps /var/log/ufw.log is empty.
And /var/log/kern.log doesn't have anything other than a stripped down version of /var/log/syslog. Which is showing nothing at time of crash, only the bootup.
 
Old 10-13-2010, 11:50 AM   #8
quanta
Member
 
Registered: Aug 2007
Location: Vietnam
Distribution: RedHat based, Debian based, Slackware, Gentoo
Posts: 724

Rep: Reputation: 100Reputation: 100
FYI: https://bugs.launchpad.net/ubuntu/+s...og/+bug/407862
 
Old 11-12-2010, 04:07 PM   #9
fsargent
LQ Newbie
 
Registered: Nov 2010
Posts: 1

Rep: Reputation: 0
Unhappy

:msg,contains,"[UFW " /var/log/ufw.log
^this line breaks things for me in rsyslog. It'll cause a syntax error when generating the rules in rsyslog. This can be viewed by running it in interactive mode.

Is there any other way I can tell UFW to output to another file, or more to the point, to a remote server (which is my ultimate goal)?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
server crashed pankajd Linux - Hardware 1 09-01-2009 04:33 PM
server is crashed rizwan.mirzaa Linux - Server 1 07-01-2008 07:33 PM
X Server Crashed a1opus Fedora 1 04-03-2006 12:08 PM
how do I determine which disk has crashed synthol6 AIX 4 12-21-2004 04:16 AM
my server crashed inteltechs Linux - General 8 11-02-2003 09:58 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 03:35 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration