Hello Folks,

We have bunch of servers with different OS versions of RHEL and SLES. Recently all our servers went to grub state all of a sudden one after the other. We just tried re-installing OS on all the machines followed by post OS installation of some tools like telnet,ftp,xinetd,etc.But all our efforts were not fruitful, all the servers again going to grub state after a day or so. we tried this experiment like 3-4 times but again observing the same behavior. Even we were not able to recover the machines from grub rescue state because of missing partitions. We tried thinking in all dimensions but still not root caused the issue. Could any one throw some light in root causing the issue and the possible issues that are hidden in this.Also i want to know in which scenarios a server goes to grub state.

Note: servers going to grub state just even after installing OS(with out post OS installation).

Thanks,
Ramakrishna.

would be nice to describe what do you mean by "all dimensions". Also what do you mean by grub state. What happened to the partitions? I think that can be a power problem, but you gave almost no information, so I can only guess.

1 members found this post helpful.

Thanks for the quick reply.

We have ethernet card(2-port) inserted in the machine. We have tried the below experiments

1)Installed OS with ethernet card in it
2)So just to eliminate the suspicion on card we have removed the card and installed the OS.
3)Changed the power cables and line.

grub means the system crashed and were you have some limited commands or means to recover. I have gone through the grub rescue steps available in different sites but couldn't able to recover the server.

Let me know if you need more info.

Thanks,
Ramakrishna

If your partitions go away, I can see three possible causes:

• Bad hardware - interfaces, cables, switches, disks, power, ....
• Something in the OS overwrites the partitions.
• Something outside overwrites the partitions. Are you sharing disks?

When a server crashes, is there any panic message on the screen?
Can you retrace the recovery steps you took and how and at what point the recovery failed?
Any other context you can share?

Not able to see any kernel panic or anything on screen, i could only see the grub screen.

i followed the steps from the below link

http://askubuntu.com/questions/11959...own-filesystem

Attached the screen shot of grub screen and steps followed in recovering.

Attached Thumbnails

   

If you have RHEL and SLES systems, you better find instructions for RHEL and SLES rather than Ubuntu, as system initialization is an area where distros differ.
By the way, there is also a big difference between RHEL6 and 7 - RHEL 6 still uses legacy grub, RHEL 7 grub2.

Here are instructions for Centos that might apply to your case: https://wiki.centos.org/TipsAndTricks/ReinstallGRUB.

I suspect that your partitions are fine, but they are not on the disk where grub looks for them. In other words, your grub installation might be wrong. Use the install disk to boot into rescue mode to check this. Any LiveCD (or USB stick) would be similarly helpful.

1 members found this post helpful.

I have mounted iso file and tried installing grub and it went fine. Then unmounted iso and did reboot. But it is again booting to rescue saying boot device not found.

Attached Thumbnails

 

that can be a simple misconfiguration, either you have disabled something in bios or you have a wrong /etc/fstab (or something similar).
Did you configure any raid related settings?

In post 5 you have two images indicating "no such partition" and unknown filesystem. Your partitions have changed for some reason and Grub is unaware of the change. Since you have only Linux filesystems, I'm surprised you get an "unknown filesystem" error. Might need fsck?? The image in post 7 indicates a UUID for a partition pointed to by Grub does not exist which could be a result of changing partitions without updating Grub. Posting more detailed information would be the best next step. If you go to the site below, download and burn the boot repair to a CD and boot with it you should be able to get detailed info by selecting the "Create Bootinfo Summary" option and posting the link. Do not try to run any repair.

https://help.ubuntu.com/community/Boot-Repair