Shutdown command no longer works - CentOS 6u6 DL380-G8

mindg4m3 · 05-22-2015, 10:46 PM

I have a set of 6 HP DL380 G8s that no longer honor shutdown -<anything>, ctrl-alt-del, or any of the /sbin/poweroff like commands
The last message displayed is:

Code:

Deactivating block devices:

These machines would shutdown fine prior installing Nvidia cards and Cuda Drivers, which are necessary for the workload.
There are no logs on the shutdown event to provide, and the last entry is releasing the above message. I believe the issue may be power control related as the disks are released cleanly after a hard cycle.
Things I have tried on the kernel line in grub:

Code:

acpi=force
acpi=off
reboot=a
reboot=a,e,k,f,c

I would like to be able to reboot these servers without having to go through ILO. Any ideas or suggestions to try would be much appreciated!

unSpawn · 05-23-2015, 05:38 AM

Have you checked what installing Nvidia + Cuda entails dependency-wise? (As in keeping tabs on everything that changed.)
How about using remote syslog / netdump?
Are there any LKMs or processes alive on that last syslog message (use sysrq and generate a task list)?
Can you reboot when you first drop to runlevel 1 and then reboot?

mindg4m3 · 05-24-2015, 01:30 AM

Oki! So I got it figured out, thanks in part to your suggestion.

Using the sysrq I was able to demonstrate that the system does in fact know how to power off the system.

What the problem ended up being was that UPSTART or the rc scripts were not interacting properly with the vncserver daemon, causing it to wait on a dead lock file.

This was resolved by creating a conf file that would force vncserver to clean up and sticking that into /etc/init to execute on runlevels 0 and 6.

Thanks for your help!

unSpawn · 05-24-2015, 05:43 AM

You're welcome.