Random Reboots (+ Linux Coding Question)

ckozler · 09-12-2011, 07:49 AM

Hi Guys,

This is a two part question- the second question was derived from the issue I am having as outlined in the first question

First question - I am running a Dell R610 server with CentOS 5.6 with a Xen kernel

Code:

[08:39:16][root@virtualmaster-01:~]$ uname -a
Linux virtualmaster-01.mydomain.com 2.6.18-238.19.1.el5xen #1 SMP Fri Jul 15 08:16:59 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

Twice now I have seen a random reboot of this server on a weekend- once on a Sunday in the early hours and then again this past Saturday at around 3 AM.

Code:

[08:40:05][root@virtualmaster-01:~]$ last -n 10 
root     pts/7        192.168.1.211    Mon Sep 12 08:39   still logged in   
{...}   
reboot   system boot  2.6.18-238.19.1. Sat Sep 10 03:57         (2+04:43)

My question is- is there any way I can find out what caused this? I see no logs about a kernel panic, there was no power failure as this server is collocated with about 8 other servers and a NAS device and all of which did not report a similar issue. This server also has dual power supplies and more recently just received a memory upgrade about 2 weeks ago- up until this point though, nothing happened.

What interests me though is that last shows it as a reboot and not as a crash. /var/log/messages* does not show that it was a command-initiated reboot (eg: reboot executed at the command line) and only shows logs from the subsequent boot

Any ideas? Anything else I could supply that could give you anymore details? I do not have automatic installation of updates going either...I do that manually.

Also should note I am the only one with access to this box, both physically and remotely.

My second question:

Could anyone point me into the direction of how to hook Linux functions and system calls? For instance, I want to write a small driver/daemon utility that will catch a reboot or a shutdown - whether user executed or a crash/system reboot like the one depicted here- so I can figure out my own way to handle it. I probably wont utilize it on any production system and I am more curious as to how to do it so I can rehash my C/C++ knowledge. Any pointers/directions or tutorials I could look at would be a great help, thanks!

zackwasa · 09-12-2011, 08:18 AM

Try this script to log thing that runs on the server each minute. I hope it will help you determine the cause:

Code:

#!/bin/sh

mkdir -p /var/log/monitor
date=`date +%H_%M`

top -b -n 1 > /var/log/monitor/top_$date
netstat -anp > /var/log/monitor/netstat_$date
ps -efww > /var/log/monitor/ps_$date
free > /var/log/monitor/free_$date
w > /var/log/monitor/w_$date

Put it as cron to run each minute

RMI

ckozler · 09-12-2011, 08:45 AM

Quote:

Originally Posted by zackwasa

Try this script to log thing that runs on the server each minute. I hope it will help you determine the cause:

Code:

#!/bin/sh

mkdir -p /var/log/monitor
date=`date +%H_%M`

top -b -n 1 > /var/log/monitor/top_$date
netstat -anp > /var/log/monitor/netstat_$date
ps -efww > /var/log/monitor/ps_$date
free > /var/log/monitor/free_$date
w > /var/log/monitor/w_$date

Put it as cron to run each minute

I was already leaning on towards implementing something like this but Im not sure this will fix the issue. I have a tiny inclination that it may be a failing/failed Xen driver or something that is calling init 6 and executing a reboot. While the chance of it being a hardware failure is there, it is a significantly small chance and I believe it may be something software related.

That being said, cronjobs may not be the answer. I have one already that defines @reboot to email me and take a snapshot of all running processes and network connections as reboot is called. This could be a temporary fix but if the system is actually crashing and its the result of a failed driver, I have a feeling cron may never get called unless it too has hooks into init.

Let me know what you think and thank you for your input!