Good Morning,

I am running RHEL 5.5:So I was running yum update yesterday on my distant servers, it had alot of updates since the last time they were updated was back in Janurary, I wasn't with company at the time. However it seem like all my servers ran just fine after my reboot. However I come in this morning and it looks like my Backup server running Veritas Software is offline cause I can't ssh to it or ping it. Would there be anything in the updates I should of been more cautious about cause it did update the kernel but like I said before all my other servers are fine up to this point. Not really looking for an answer just maybe some suggestions to be more prudent the next time I run a yum update.

I saw a problem at a previous job with a system that hadn't been patched in a while ... the root partition and it's mounts weren't fsck'ed in forever (hadn't been rebooted in 1.5+ years or so) and it forced a file system check. Needless to say there were a ton of errors it flagged, and the OS didn't boot anymore. I spent a while trying to get it back online, but we ended up just reinstalling the base OS and mounting all of the applications back again (Oracle Collaboration Suite). Luckily we had the apps on a separate partition/drive, so the application was safe. That was when I realized it was good practice to keep apps on a separate partition, or better yet, separate mirrored drive. Otherwise we would have had to call up some Oracle DBAs to help us reinstall the app, too.

After that, we were required to started patching servers on a monthly basis and I never saw the issue again.

Other than that, if you are going to patch or reboot a high visibility production server, make sure you have some way of remote console to the system (KVM, DRAC, etc).

- Edit for clarity.

1 members found this post helpful.

Ok I got 2 more servers that crashed, my follow up question is would a kernel update make it where the drives wont even light up, shouldn't I at least be able to get through the BIOS before it crashes on the OS, because they won't even start booting.

A kernel update is in fact a new kernel entry added to /boot/grub/grub.conf. It doesn't actually update (ie use new kernel) the running kernel unless you reboot the machine.
If you have rebooted, then maybe you need some updated drivers in the initrd and/or modeprobe.conf.
These links may be handy
http://www.linuxtopia.org/online_boo...ion/index.html
http://www.linuxtopia.org/online_boo...ide/index.html

I've heard some horror stories about kernel updates ... not in Linux so much, but on the Unix side (either AIX or Solaris, I don't recall exactly). There was a situation where a certain type of RAID controller or storage driver became obsolete, and instead of providing backward compatibility, the vendor simply removed the storage adapter code completely from the kernel, causing all kinds of issues.

chrism01's post may be of use to you. Best bet is to attempt to boot into the old kernel, check all of the modules that were being loaded, and make sure those are still being loaded correctly in the new kernel.

Good luck, and let us know for everyone's benefit what the solution was if you find it.

Well it looks like the Kernel Update was not the problem after all, I'm in Texas and some of my Servers are in the East Coast and what ended up happening was the UPS was going bad, forcing my Sun Servers to go into Standby mode and not being able to come back up till the servers saw the right voltage. It just so happens I was running these updates as this was coming to fruit.I'm glad this came to light cause I've only been here 3 months and I wasn't looking to good in front of my bosses.

I suspected something like that; yum doesn't change what's running, just what's on disk. Other progs may get cold started (re-loaded from disk) in the normal flow of things, but not the kernel.