Alright, this is one of the diciest problems I've ever encountered, so don't feel bad if you don't have a clue on how to help.

The back-story is that I've got a root-access server through 1and1. I've had it for quite some time, so it's based on FC5, I think.

A few days ago, it had a problem with the root filesystem and, as a result, upon reboot, fsck threw everything into lost+found. (/var, /usr, and /home were in other partitions, so they're intact... but /bin, /sbin, /etc, /root, /boot, /proc, /sys, /opt, etc. were all tossed into lost+found).

So, the server wouldn't boot like this, of course, so I used the admin web page at 1and1 to boot into the "recovery system", which is Debian-based.

I was able to mount the root partition and identify the lost/found directories by their contents and move them back into the root. Also, from all indications, the files seem intact (ie, the files in /etc look to have the proper contents, running "file" on the files in /sbin and /bin gives expected results, etc).

However, the system *still* wouldn't boot. GRUB was able to find the kernel and start loading it (a step forward), but then it would stop when it said it couldn't find "init" and would reboot.

I'm almost certain that it could *see* init (which was located in /sbin), but that it just couldn't execute it. Running "file" on init yielded:

ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.6.9, dynamically linked (uses shared libs), stripped

So, that looks fine. But I was clued into a problem when I tried to use chroot to sandbox my shell into the filesystem I was trying to recover (remember, I'm doing this from the rescue environment, and I had the "normal" root filesystem mounted at /mnt). When I tried "chroot /mnt", I'd get a bunch of messages about "/bin/fgrep: No such file or directory" and "/sbin/consoletype: No such file or directory". Those files are there, and "file" reports that they're ELF executables. But I was wondering if the shared libraries were a problem...

So, I used Ermine to create a statically-linked version of init from a Debian box that I administer, and I put that in place of the init on the box I'm recovering. I rebooted, and the kernel no longer complained about not being able to find "init". Unfortunately, it *hung* at the place where it tried to run init, but I think that I'm making headway.

I feel like my main problem is that all of these binaries which are needed for the initial system startup and use dynamic linking (something I'll never understand... why these few mission-critical binaries aren't statically linked) aren't able to find their libraries.

So, I guess my primary question is: If "init" is dynamically linked, and it gets run before anything else which can set up paths, then how do I know where it's looking when the kernel tries to run it?

Some other questions I have which I'm hoping to get answers to are:
1) Is there some way to check all of these files to make sure that they're not corrupt? I guess rpm has a "-V" option, but I have the problem that my recovery environment is Debian-based and also that I don't have the FC5 rpm's on the machine.
2) Is there some trick anyone can offer for being able to minimally boot the system? Perhaps replacing init with a basic statically-linked shell or something?

Thanks in advance.

Oh, by thew way, yes... re-imaging the system is an option, but it's the LAST option. We've done a lot of configuration of this server so, even though we've got all of our user files backed up, we don't have all of the Plesk configuration stuff backed up, so we'd really like to not have to configure all of that stuff again.

Try to pass the kernel "init=/bin/sh".

I wouldn't make the problem relate to the lib files for sure. Try to think at other areas too.


Don't forget to report back to us, this is a very intresting problem.

Be aware that in case it is a X86-64 system, you have to have 32bit & 64bit libs.
Check with file what bit is your init or whatever isn't run as should.

Check the LD_LIBRARY_PATH env and see if it contains the right paths to the your libs.
Chech with ldd on the init program to see if it has the required libs, although i am in doubt if it work because you are using a rescue OS, so try to chroot.

Hope it helps.