We have a xenserver 5.6 machine that suffered through a power outage. Now, xsconsole loads but hangs repeatedly. I have to press ^C several times to manage to break out into the CLI.

Can't start any vms from the CLI either. Tried restarting the toolstack with xe-toolstack-restart and then it just spins forever at stopping xapi.

Tried to stop and restart xapi through service xapi restart (or stop) and the same thing.

Forget trying to connect from XenCenter or OpenXenCenter. Keeps on trying to synchronize.

From dmesg, I found:

Also, at the start of dmesg there is an insane number of
"ext3_orphan_cleanup: deleting unreferenced inode #####" where #### is a number but they are all in reverse sequence starting from 377344 down to 376838 (975)


"INFO: task xenstore-read:7881 blocked for more than 120 seconds"

This message continues for about a 2 dozen or so instances before the end of the file.


Trying to reboot and I noticed the following:

Shutting down Virtual Machines, Unable to connect to master within 30 sconds. Exiting. [FAILED]

Then

"Terminating remaining virtual machines" which also fails for the same reason.

Then

Storing network configuration then just hangs..

Any ideas how I can recover?

You need to check your host filesystem, dude! "fsck", or moral equivalent.

This link might help:
http://support.citrix.com/article/CTX116113