fsck root file system
 

I have two identical servers. Then the 1st server got an issue. Last action done was it was fsck. My suspect is that / file system has been fsck while it was mounted. I'm not sure but at least we know the last action done. Now, it is experiencing kernel panic. I already boot to rescue cd and was able to mount the /boot and / of the server but still experiencing the same issue.

Has anyone could provide advise how to restore? I'm thinking to copy /boot from the replicated server via rsync but I'm not sure if this is the correct strategy. I have already fsck / and /boot and has no issue.

First thing is make sure that the disk is physically healthy ... I'd suggest running the short and long online self tests via smartctl (e.g. "smartctl -t short /dev/sda" -- the smartctl man page has more information). If SMART reports errors then probably the disk is on its way out and needs to be replaced. Note that fsck will generally refuse to run on a mounted filesystem (unless it's mounted read only). Why do you suspect the fsck was run while the root filesystem was mounted?

In principle you could use dd or similar to do a bitwise copy of the second server's hard drive onto the first server's disk (you'd have to have them both in the same machine and be very careful not to reverse the if and of arguments to dd). If these are production servers, though, why on earth don't you have back-ups already that you can just restore from?

It doesn't sound like you're particularly confident about the source of the problem, though, so for me step 1 would be getting a bit more certainty about that before trying to fix it.

I am just booted on rescue cd. It could not load the root disk. It always in kernel panic. I could not restore from backup.

If you want help you'll have to provide enough detail as to what you did and what failed. All of the following are useless