Major problem with software raid (mdadm) and disk failure
 

Ok, this is the case:

I've got two raid-5 arrays with software raid, both with three disks.

Setup:
md0 has hdb2, hdd1 and sda1
md1 has hdb5, hdd3 and sda3

Tonight, the system lost power due to a power spike. The result was a reboot where it attempted to fix the raid, but it didn't exactly work. I have now booted a live CD and using utilities there.

It seems the checksum value is slightly off for all disks. And the update time is the same for some disks, but not all.

When I try mdadm --assemble /dev/md1 (or md0) with various options like force, scan, auto, etc - I get error messages like only 1 drive found, not enough.. no disks found and no mention of md0/1 found in config file.

What should I do? Which partitions (two in each array) do I pick as "these two are the one I should rebuild with"? And how do I actually FORCE mdadm to start an array with those two set devices?

See files in ... for output from mdadm.

Assistance deeply appreciated...

ooops... the URL for the files: http://gandalf.n4f.no/~aj/md/

Hi norwolf,

md1 should be startable from just hdb5 and sda3, as those two still match. Once it is up, let it rebuild hdd3.

md0 will be trickier, because none of the disks match. hdd1 is the worst (oldest) so that's the one you want to start without and rebuild later, once you have the array going with just hdb2 and sda1.
I don't know if mdadm will let you assemble the array with hdb2 and sda1, maybe with some force/run persuasion. Worst case would be to actually re-create the array from those two disks.

well, I haven't found a command that will do that yet. I've tried mdadm --assemble /dev/md1 --force, mdadm --assemble /dev/md1 --scan, etc. Any clues?

Have you tried explicitly giving the device names of the partitions you want to add?
I can't test it right now, but something like Then add --force and --run as needed. ;o)

Good news after a quick test: it should work painlessly as suggested.
For md1:
For md0:
(And obviously do a fsck afterwards, especially on md0)

Tried the following:
# mdadm --assemble /dev/md1 /dev/hdb5 /dev/sda3
no devices found for /dev/md1
# mdadm --assemble --force /dev/md1 /dev/hdb5 /dev/sda3
no devices found for /dev/md1
# mdadm --assemble --run /dev/md1 /dev/hdb5 /dev/sda3
no devices found for /dev/md1
# mdadm --assemble --force --run /dev/md1 /dev/hdb5 /dev/sda3
no devices found for /dev/md1

Although - the devices ARE there.

hm, intersting.
Try adding a '-v' switch on the assemble, maybe that'll give some useful info.

PS: Is there a /etc/mdadm/mdadm.conf on the system you are trying to assemble from (i.e. the live cd)? If so, please post the contents.

-v didn't give any more info.

No, there isn't an /etc/mdadm.conf or /etc/mdadm/mdadm.conf. The entire system is offline, we've got it booting from a rescue CD.