LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Server (http://www.linuxquestions.org/questions/linux-server-73/)
-   -   Major problem with software raid (mdadm) and disk failure (http://www.linuxquestions.org/questions/linux-server-73/major-problem-with-software-raid-mdadm-and-disk-failure-572596/)

norwolf 07-26-2007 08:31 PM

Major problem with software raid (mdadm) and disk failure
 
Ok, this is the case:

I've got two raid-5 arrays with software raid, both with three disks.

Setup:
md0 has hdb2, hdd1 and sda1
md1 has hdb5, hdd3 and sda3

Tonight, the system lost power due to a power spike. The result was a reboot where it attempted to fix the raid, but it didn't exactly work. I have now booted a live CD and using utilities there.

It seems the checksum value is slightly off for all disks. And the update time is the same for some disks, but not all.

When I try mdadm --assemble /dev/md1 (or md0) with various options like force, scan, auto, etc - I get error messages like only 1 drive found, not enough.. no disks found and no mention of md0/1 found in config file.

What should I do? Which partitions (two in each array) do I pick as "these two are the one I should rebuild with"? And how do I actually FORCE mdadm to start an array with those two set devices?

See files in ... for output from mdadm.

Assistance deeply appreciated...

norwolf 07-26-2007 08:32 PM

ooops... the URL for the files: http://gandalf.n4f.no/~aj/md/

rupertwh 07-26-2007 09:33 PM

Hi norwolf,

md1 should be startable from just hdb5 and sda3, as those two still match. Once it is up, let it rebuild hdd3.

md0 will be trickier, because none of the disks match. hdd1 is the worst (oldest) so that's the one you want to start without and rebuild later, once you have the array going with just hdb2 and sda1.
I don't know if mdadm will let you assemble the array with hdb2 and sda1, maybe with some force/run persuasion. Worst case would be to actually re-create the array from those two disks.

norwolf 07-26-2007 10:23 PM

well, I haven't found a command that will do that yet. I've tried mdadm --assemble /dev/md1 --force, mdadm --assemble /dev/md1 --scan, etc. Any clues?

rupertwh 07-26-2007 10:39 PM

Have you tried explicitly giving the device names of the partitions you want to add?
I can't test it right now, but something like
Code:

mdadm --assemble /dev/md1 /dev/hdb5 /dev/sda3
Then add --force and --run as needed. ;o)

rupertwh 07-26-2007 11:33 PM

Good news after a quick test: it should work painlessly as suggested.
For md1:
Code:

mdadm --assemble /dev/md1 /dev/hdb5 /dev/sda3
mdadm /dev/md1 --add /dev/hdd3

For md0:
Code:

mdadm --assemble /dev/md0 /dev/hdb2 /dev/sda1 --force
mdadm /dev/md0 --add /dev/hdd1

(And obviously do a fsck afterwards, especially on md0)

norwolf 07-27-2007 04:27 AM

Quote:

Originally Posted by rupertwh
Have you tried explicitly giving the device names of the partitions you want to add?
I can't test it right now, but something like
Code:

mdadm --assemble /dev/md1 /dev/hdb5 /dev/sda3
Then add --force and --run as needed. ;o)

Tried the following:
# mdadm --assemble /dev/md1 /dev/hdb5 /dev/sda3
no devices found for /dev/md1
# mdadm --assemble --force /dev/md1 /dev/hdb5 /dev/sda3
no devices found for /dev/md1
# mdadm --assemble --run /dev/md1 /dev/hdb5 /dev/sda3
no devices found for /dev/md1
# mdadm --assemble --force --run /dev/md1 /dev/hdb5 /dev/sda3
no devices found for /dev/md1

Although - the devices ARE there.

rupertwh 07-27-2007 05:45 AM

hm, intersting.
Try adding a '-v' switch on the assemble, maybe that'll give some useful info.

PS: Is there a /etc/mdadm/mdadm.conf on the system you are trying to assemble from (i.e. the live cd)? If so, please post the contents.

norwolf 07-27-2007 06:14 AM

Quote:

Originally Posted by rupertwh
hm, intersting.
Try adding a '-v' switch on the assemble, maybe that'll give some useful info.

PS: Is there a /etc/mdadm/mdadm.conf on the system you are trying to assemble from (i.e. the live cd)? If so, please post the contents.

-v didn't give any more info.

No, there isn't an /etc/mdadm.conf or /etc/mdadm/mdadm.conf. The entire system is offline, we've got it booting from a rescue CD.


All times are GMT -5. The time now is 04:24 AM.