Help! Reboot Fails aft Debian Etch upgrade now at GRUB prompt
 

Help!! I'm a debian server admin who is preparing to upgrade from etch to Lenny and was told I should upgrade from Etch from last August to the final release of Etch from May '09 first.

So, I made a full boot drive backup with dd. Then I ran the "mock upgrade" using aptitude first and everything went well. Then I ran the real upgrade (128 pacakges were upgraded) and everything seemed to go well too. (Famous last words!)

In the middle of the upgrade I got a notice that I'd need to reboot after the upgrade had finished in order to let Apache complete its rebuild (No surprise there!). So when the upgrade completed successfully with no horrible errors or warnings, I double-checked to make sure all updates had been installed and then I tried to reboot the system. ARggghhhhh! The bleeping reboot failed.

The server is remote (1,200 miles away). So I requested KVM access from the server center and find I'm at the GRUB loader's prompt...

YIKES! I have NO idea what went wrong or what the exact name of the system bootstrap file is. I do know it's on sda1 (hd0,0) in Grub's parlance) and I know my full save (dd) backup is on sdb1 (Is that hd1,0 to grub?). I also know the Etch release I was running ended in ".18". I'm fairly sure it was like vmlinuz-2.6.18 and that it ended with -amd64.

What the heck do I do NOW??

if you can over KVM boot to some live CD system and see what is in /boot, what kind of kernel is placed there during upgrade.

Then try to compare it with value in menu.lst, what values for kernel boot are there, still old ones, or ....

What kind of error you see ?

This link http://www.mondorescue.org/ describes mondo backup solution....I recomend it, do backup, test does backup works--if yes, perfom upgrades/changes



Regards

This server center gives very little support. The only set of boot CDs they are likely to have there would be the original OS install CD's and they might well be for Lenny now and not for Etch. Is it possible get into recovery mode from those CDs? Clearly, I don't want to do a fresh OS install! What I need to do now is get a good look at the root directory on that boot drive and on the backup drive as well.

I will also look at the mondorescue.org site. But I assumed the full-drive dd copy would produce a bootable backup. In fact, I'm not sure it DIDN'T produce a bootable backup. I'm just not clear about how to convince GRUB to boot from that drive.

Confucious say: You never know what you should know until you need to know it! :-(

Thanks.

yes.

You are getting as far of the grub prompt so that is good news.

Here is a pointer to the GRUB manual. You are going to need it as a reference.

What is the output of the following grub command?

Well, having burrowed deep into the manure pile that is grub manual in the past, I had hoped to avoid that horrid experience again. But I did manage to find a great grub tutorial (I KNOW that's a contradiction in terms, but it's true.) at:

http://www.troubleshooters.com/linux/grub/grub.htm

This helped me figure out the location and name of the vmlinuz file; but when I tried to reboot using that information (from hd0 which is actually sda1) I got the following kernel panic:

Before getting this error, I used these commands in grub:

I also tried this again with the following commands

And it produced the same kernel panic error with a reference to sda1 rather than hda1

Does anyone know what I'm doing wrong with the device names?

I forgot on the restart to look up the output of the "cat /grub/grub.conf" command. I'll try that on this next restart. And I gues I'll have to go dig out that grub manual after all... sigh!

Thanks again!

This command doesn't seem to work. When I try it all I get is "File Not Found"

I've also tried

find /grub/gru<tab>

in an effort to use autocomplete to figure out the file name. But it isn't working either. This IS a debian box. That MAY mean that the file name for grub.conf has changed here.

Is there another way to locate it?

If you're looking for the configuration file it should be menu.lst which is in the Grub directory so your command should be:

cat /boot/grub/menu.lst

Your vmlinuz file should be in the /boot directory.

Yes, you are right. I was thinking of Fedora/Redhat/Centos for some reason.

The following will not work until you add details of your initial ram disk using initrd.

You're right, Yanek. The grub directory was inside the boot directory. I knew it was near the top but wrongly remembered it as being in the root directory. That's why I couldn't find it. I figured that out later... shortly before I discovered my server has been rootkit-ed...

So now I have a totally different batch of fish to fry. :-(

Thanks for the help!

Yes, fpmurphy. You were right. The problem I was having was I was missing the need for the initrd command.

I'm sure I read somewhere (I believe it was in this excellent grub tutorial: http://www.troubleshooters.com/linux/grub/grub.htm) that in many cases the initrd command wasn't needed after it had been provided once. Yet in my case, the system repeatedly fails to boot if this grub command is NOT included.

It clearly wasn't required for the old kernel to boot.

Does anyone know of a way to get the info provided in initrd embedded into the kernel or must I always include it in the bootup from this day forward. If so, can someone explain why?

For the moment, I'm ignoring the fact that I'm still trying to decide what to do about that &^%$#@! rootkit and want to complete my knowledge on how to eliminate the kernel panic caused by failing to include the initrd command in grub.

Thanks!

You can build a kernel that does not need an initrd to boot by compiling all the modules into the kernel but most people do not bother. Doing so eliminates a certain amount of splash functionaility and just makes it harder to do kernel upgrades since you will have to roll your own kernel each time.

Assume than you always need an initrd line in your (grub) configuration stanza.

Thanks a bunch, fpmurphy. Am I correct in concluding that compiling a new kernel also automatically produces a matching intrd image file too? Is that the deal? That's what seems to be the case. I'm just confirming my gradually growing understanding...

Thanks again!

If you build a kernel RPM, yes. If you just build a kernel, no.