Why is my server reboot

packets · 08-27-2017, 12:53 AM

I'm trying to investigate why the server rebooted automatically. First thing I check is the last command. I notice the logs below.

Code:

reboot   system boot  3.10.0-514.26.2. Fri Aug 25 08:49 - 13:46 (2+04:56)  
reboot   system boot  3.10.0-514.26.2. Tue Aug 22 11:02 - 13:46 (5+02:43)

Checking the above shows the server was not rebooted by anyone because on the third column the source location is the kernel. Is this correct?

TB0ne · 08-27-2017, 10:24 AM

Quote:

Originally Posted by packets

I'm trying to investigate why the server rebooted automatically. First thing I check is the last command. I notice the logs below.

Code:

reboot   system boot  3.10.0-514.26.2. Fri Aug 25 08:49 - 13:46 (2+04:56)  
reboot   system boot  3.10.0-514.26.2. Tue Aug 22 11:02 - 13:46 (5+02:43)

Checking the above shows the server was not rebooted by anyone because on the third column the source location is the kernel. Is this correct?

Read the "Question Guidelines" link in my posting signature. You don't tell us what version/distro of Linux, and that certainly isn't the entire output of the last command. Read the man page on last.

All that shows is the system boot...there are most certainly OTHER entries there that can tell you more. Since you tell us nothing about what you've looked at, we can't help.

jsbjsb001 · 08-27-2017, 10:47 AM

Quote:

Originally Posted by packets

I'm trying to investigate why the server rebooted automatically. First thing I check is the last command. I notice the logs below.

Code:

reboot   system boot  3.10.0-514.26.2. Fri Aug 25 08:49 - 13:46 (2+04:56)  
reboot   system boot  3.10.0-514.26.2. Tue Aug 22 11:02 - 13:46 (5+02:43)

Checking the above shows the server was not rebooted by anyone because on the third column the source location is the kernel. Is this correct?

While I totally agree with TB0ne, are your using CentOS?

packets · 08-29-2017, 09:00 PM

Sorry for the incomplete details. OS is CentOS Linux release 7.3.

Nothing found in the /var/log/messages prior on the reboot.

Code:

Aug 25 00:40:02 test systemd: Removed slice user-0.slice.
Aug 25 00:40:02 test systemd: Stopping user-0.slice.
Aug 25 08:49:25 test rsyslogd: [origin software="rsyslogd" swVersion="7.4.7" x-pid="777" x-info="http://www.rsyslog.com"] start
Aug 25 08:49:25 test rsyslogd-2307: warning: ~ action is deprecated, consider using the 'stop' statement instead [try http://www.rsyslog.com/e/2307 ]
Aug 25 08:49:08 test kernel: Initializing cgroup subsys cpuset
Aug 25 08:49:08 test kernel: Initializing cgroup subsys cpu
Aug 25 08:49:08 test kernel: Initializing cgroup subsys cpuacct
Aug 25 08:49:08 test kernel: Linux version 3.10.0-514.26.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Tue 
Jul 4 15:04:05 UTC 2017
Aug 25 08:49:08 test kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.26.2.el7.x86_64 root=UUID=ef6ba050-6cdc-416a-9380-c14304d0d206 ro console=tty0 console=ttyS0,115200n8 crashkernel=auto console=ttyS0,115200 LANG=en_US.UTF-8
Aug 25 08:49:08 test kernel: e820: BIOS-provided physical RAM map:

MadeInGermany · 08-30-2017, 04:19 AM

Maybe a hardware problem?
Is it a server hardware? Then there might be a HW fault log in the server management console (ILO board or similar).

jlinkels · 08-30-2017, 07:17 AM

Mystery reboots are often caused by hardware problems.

Reboots occurring at exactly the same time can be induced by a cron job. The running process can either cause a higher processor load (thermal stresses) or uses a part of the physical memory not normally used.

It can takes days if not weeks before the hardware finally fails to a greater extent making it easier to diagnose.

If it occurs at the same time and therefor one of the cron jobs is suspected, the job can be run manually to if that triggers the reboot.

Unfortunately it still doesn't tell if the process is responsible for the reboot or the hardware. However, reboot without any message from the running process is an indication for a hardware failure. The only way to know for sure is to migrate to new hardware.

jlinkels

businesscat · 08-30-2017, 07:39 AM

could you show us a cron job?