How to troubleshoot problem with system not actually rebooting after shutdown command

Arodef · 09-12-2014, 12:22 PM

I'm trying to remotely reboot a RHEL 5.3 server I have SSH access to.

I issued the reboot command and I got the 'The system is going down for reboot NOW!' message but it never actually rebooted. I did it multiple times since it was still up, I expected to be booted off SSH connection but it stayed up:

Code:

Sep 12 08:33:07 sblade1 shutdown[29489]: shutting down for system reboot
Sep 12 08:33:10 sblade1 init: Switching to runlevel: 6
Sep 12 08:37:43 sblade1 shutdown[31275]: shutting down for system reboot
Sep 12 08:38:49 sblade1 shutdown[31694]: shutting down for system reboot

It's now a couple hours later and the system is still up and I can SSH to it still. Same behavior from shutdown -r now command. How do I figure out what's preventing the reboot?

My hunch is the actual terminal on the server shows the progress from the initial shutdown I initiated, however something might be blocking a step and it's stuck at that particular step of the shutdown process? I only have SSH access so I don't know how I could see that remotely.

Any suggestions?

Arodef · 09-12-2014, 02:57 PM

Update: The monitor attached just showed a login screen. I had someone do a hard reset of the server. It came back up. From examining dmesg, it shows a message about 63 orphan inodes deleted from an attached external drive (see below). Perhaps that filesystem could not be unmounted so that's what was preventing the reboot? Server seems to be running fine though so I'll keep my fingers crossed.

Code:

EXT3-fs warning: checktime reached, running e2fsck is recommended
EXT3 FS on dm-2, internal journal
ext3_orphan_cleanup: deleting unreferenced inode 330301458
ext3_orphan_cleanup: deleting unreferenced inode 330301462
ext3_orphan_cleanup: deleting unreferenced inode 330301470
ext3_orphan_cleanup: deleting unreferenced inode 330301473
ext3_orphan_cleanup: deleting unreferenced inode 330301457
ext3_orphan_cleanup: deleting unreferenced inode 556910141
..SNIP...
ext3_orphan_cleanup: deleting unreferenced inode 556910093
ext3_orphan_cleanup: deleting unreferenced inode 556910087
ext3_orphan_cleanup: deleting unreferenced inode 556922666
ext3_orphan_cleanup: deleting unreferenced inode 556910089
ext3_orphan_cleanup: deleting unreferenced inode 556910084
EXT3-fs: dm-2: 63 orphan inodes deleted
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.