Hello all,

2 nodes in a redhat cluster have gone down for unknown reasons. while trying to reboot I get the following:

fs error ext3_check_descriptors
group descriptors corrupted
error 22 mounting ext3
pivot_root(/sysroot, /sysroot/inited failed:2)
kernel panic no init found try passing init option to kernel

so while booting when i see the grub splash i press ESC to try to edit the kernel and path init=3 however pressing "e" key doesn't do anything. only option is ctrl+X to go to txt mode. so I tried to reboot with the 3 different kernel on the server, inherit the servers recently from someone else, i try to pass init=3 however i still get the same errors.

if someone please point me to where i should be looking to get this issue solved.
i have looked at "http://www.redhat.com/docs/manuals/linux/RHL-7.2-Manual/custom-guide/rescuemode.html"
but didn't work either "couldn't mount filesystem in rescue mode"

Thank you in advance

Well, you certainly can't expect to find a "cookbook" solution to this problem: you're going to have to deduce what has gone wrong. Let me try to help...

When you say that "two nodes have gone down 'for unknown reasons,'" the first thing to do must be to try to guess what those 'reasons' could be. Did we have a power-hit? Might a disk-drive have failed?

When you see a message like fs_error_blah... it certainly sounds pretty darned plausible that a drive might have gone south for the winter. And so you might need to try booting-up one of these machines from a CD-ROM (such as a handy Knoppix distro) to see if you can make sense of what the disk-drives now contain.

Knoppix, as you may or may not yet know, is a stand-alone Linux distro that can boot entirely from a CD-ROM. It will search for your drives and mount them, if it can, but none of those drives will be "your system drive," which means that you can do what you otherwise could not do: you can diagnose your hard-drives, mount them or dismount them or repair them or what-have-you.

Knoppix is pretty-good at doing diagnostics on its own at startup, but eventually it will drop you to a root-prompt. Start by surfing Google for what they have to say about Knoppix, and about tools like fsck and smartctl.

The first thing I'd do, probably, is to use SMART to diagnose the drive. Hard-drives have on-board diagnostics. When the drive encounters an error, it records it. SMART can find and print-out those logs. If the drive's toast, "hope you got a backup."

The second thing will be to analyze and try to repair the filesystem. Once again, start by surfing Google (and this site).

Thanks for your reply and I am sorry if i came across as "demanding help or instant solution to my problems", i never meant it to be this way.

I know about Knoppix but never had to use it to diagnose system issues but i will def google-it to see how could I use it.


again thanks for your help and the advise.

Here is what I did - just in case someone else have similar issue.
booting with knoppix and ran fsck which always came back with error messages "file system block error descriptors" not exact message.

so using SystemRescueCd i booted one of the other nodes then used "partimage" and cloned the box as well as made a backup of the mbr and partaion table.

went back on the broken box and restored the images, since the two nodes are identical, and the mbr as well as the partation table.

created the swap "mkswap"

mounted the file sys and edited the hosts file to reflect the server name and edited the ifcfg-ethx to the desired IP

booted the machine and it worked.


Thank you "sundialsvcs", with out your comments I wasn't going to be thinking about liveCDs or fsck or all others points u mentioned in your post.

Later