Boot fails on ext3 Journal Recovery

Complicated Disaster · 10-19-2007, 04:40 AM

Dear Experts,

Last night I had a power failure and, when attempting to reboot my system, the boot failed. All the OS partitions checked out OK but the data partition (software RAID 1) failed....

It said that MD0 was started with one device out of two - this seems to be normal after a power failure and I just hot add the other disk back into the array when it boots. hdg1 always adds OK but hde1 is always booted. But this time is said "Recovering Journal" and then stopped. I left it overnight, but no further progress was made.

I am thinking that if hdg is knackered I can try assembling the array using just hde. But I don't know how to get the system to boot!!!

CD

bigrigdriver · 10-19-2007, 03:16 PM

http://tldp.org/HOWTO/Software-RAID-0.4x-HOWTO-4.html

Quote:

4. Error Recovery

Q: I have a RAID-1 (mirroring) setup, and lost power while there was disk activity. Now what do I do?
A: The redundancy of RAID levels is designed to protect against a disk failure, not against a power failure. There are several ways to recover from this situation.
Method (1): Use the raid tools. These can be used to sync the raid arrays. They do not fix file-system damage; after the raid arrays are sync'ed, then the file-system still has to be fixed with fsck. Raid arrays can be checked with ckraid /etc/raid1.conf (for RAID-1, else, /etc/raid5.conf, etc.) Calling ckraid /etc/raid1.conf --fix will pick one of the disks in the array (usually the first), and use that as the master copy, and copy its blocks to the others in the mirror. To designate which of the disks should be used as the master, you can use the --force-source flag: for example, ckraid /etc/raid1.conf --fix --force-source /dev/hdc3 The ckraid command can be safely run without the --fix option to verify the inactive RAID array without making any changes. When you are comfortable with the proposed changes, supply the --fix option.
Method (2): Paranoid, time-consuming, not much better than the first way. Lets assume a two-disk RAID-1 array, consisting of partitions /dev/hda3 and /dev/hdc3. You can try the following:
fsck /dev/hda3
fsck /dev/hdc3
decide which of the two partitions had fewer errors, or were more easily recovered, or recovered the data that you wanted. Pick one, either one, to be your new ``master'' copy. Say you picked /dev/hdc3.
dd if=/dev/hdc3 of=/dev/hda3
mkraid raid1.conf -f --only-superblock
Instead of the last two steps, you can instead run ckraid /etc/raid1.conf --fix --force-source /dev/hdc3 which should be a bit faster.
Method (3): Lazy man's version of above. If you don't want to wait for long fsck's to complete, it is perfectly fine to skip the first three steps above, and move directly to the last two steps. Just be sure to run fsck /dev/md0 after you are done. Method (3) is actually just method (1) in disguise.
In any case, the above steps will only sync up the raid arrays. The file system probably needs fixing as well: for this, fsck needs to be run on the active, unmounted md device.

With a three-disk RAID-1 array, there are more possibilities, such as using two disks to ''vote'' a majority answer. Tools to automate this do not currently (September 97) exist.

Hope this helps.

Complicated Disaster · 10-19-2007, 05:56 PM

Ooops. I think I should mention that I used MDADM to create the array....

CD