Hi guys.

Today happened something really weird in my redhat 5.10.

I restarted the server remotely, and after sometime it didnt come back. so when I opened the console it shows the message:

of course I couldnt login with the root account or any other account.

1- first I ran e2fschk -fy for my LVM partitions, it found some errors in my root partition(LVM) and"fixed it". didnt work, same issue at boot.

2- using google I found it might by sometime with the Xorg and advise to edit the inittab and change it to start in runlevel 3 instead of 5. So I did in rescue mode, and nothing. Now I didnt see any error message, just go to the prompt:



3- so I started in rescue mode again. automatically mount the system file.

but when I ran #chroot /mnt/sysimage I got the error:


as I was connected using a KVM from my computer I recorded my screen and this was the error I get just before "the (none) login:"




Then I boot in rescue again and when I go to /mnt/sysimage/bin it is TOTALLY empty... there are folders in the rest of folders (but not all of them)

I tried copying the /bin folder from another server, same installation (cluster and same server model)

but of course it wont boot, there are some errors from missing files, rc.d....etc etc...


So the thing is, is this possible to happen if the file system get corrupted?...we have been restarting the server several times a day while configuring the cluster.

I checked the logs in rescue mode and didnt find anything weird in the .bash_history, secure, audit.

Running fsck with the "-y" option without knowing what the problem was is a good way to lose a lot of data. You will probably find that the missing files have been dropped into lost+found, but without their original names. As for what caused it, that "fsck -y" has obliterated that trail. As an example, though, a corrupted root inode can cause an entire filesystem to get dumped into lost+found (thankfully with only the top level directory names lost).

I'm pretty sure it's an aberration. And as far as why it got obliterated in the first place, inode confusion happens when something is in memory but doesn't get written to the filesystem so the filesystem is not how the OS expects it to be. I haven't ever seen that happen to a whole filesystem though.

Did you re-install?