Recovering a Raid 5 array, mdadm mess-up
 

I've been searching through this site for some raid answers, but found nothing specific to my problem. This is my first post, so here goes :)

I have a debian Etch server, and my /home partition is set up as a RAID 5 array, with four SATA 250GB disks (750GB total). I recently returned from vacation and found that the machine was locked. After rebooting, /home did not mount. Here's what showed in syslog:

So, it seemed that two out of the four disks were failed. I was hoping that the drives overheated, perhaps the machine was not cleanly rebooted, etc. Two drives out of four drive raid5 set is not good.

I captured the output of mdadm --examine for all the disks:

Notice that the different disks had a different idea of what the state of the array was. I hoped that at worst, there was only one faulty disk.

I decided from the above output that I should try to reassemble the array. In the past, mdadm was pretty smart about trying to resync the disks. However, I made a big mistake. I typed the following command:

So, madam took a long time to rebuild the array, and then I could not mount it. I tried to reboot, no help. Here's the error from mount:

Looking at /proc/mdstat:

In horror, I realized that mdadm had built the array using the whole disks, instead of partitions. I wanted /dev/sda1, /dev/sdb1, etc ... NOT /dev/sda, /dev/sdb, etc!

Here's where I get really confused. If I look at the disks with fdisk, the partitions are still there, but two of them are just regular linux partitions (not raid autodetect):

But it gets even stranger... I no longer see the partitions in /dev:

And when I try to assemble the array now, mdadm can't find those old partitions:

So, I'm in a real bind. I don't know if my data is still on the drives (and of course, I REALLY want to recover it, only some of it is backed up). I can't see the old partitions on the drives, despite the fact that fdisk does see something.

Is it possible that my mdadm --create command wiped my disks somehow? I though mdadm was careful to check for existing raid partitions!

Any help would be greatly appreciated!

By running a create on an existing array, you've destroyed the superblocks; mdadm warns you about the existing contents of the drives when you run the command. Restore whatever data you have from backup.

This has been said many times, but it bears repeating: RAID is not a substitute for backup. It's intended to increase uptime (data availability), and does not provide data archiving.

Oh Crap
 

Wow, this sucks. The thing is, I did not get any warning, because I specified the wrong partitions (eg, /dev/sda instead of /dev/sda1). Is there really no way to reconstruct this array now? I can see raid partitions at /dev/sd[a-d]1 -- but I can't access them as the are not in /dev ... can anyone suggest something to try?

the power of mdadm
 

I did something like that once.
I just created new reiserfs on the raid discs.
Then I rebuild everything by reiserfsck --rebuild-tree --scan-whole-partition.

I recovered almost 100% of the files, and some older versions of them, too.
@500,000 files, a bit confusing, but in combination with a backup (that you restore OVER the recovered, data) your recovery-rate might be pretty good!

f

And don't make the mistake of doing software RAID on something that is important. A hardware RAID card will cost about $300. I've had almost the identical setup as you, 4 250 GB satas in RAID 5, but with a 3com controller card running it. Obviously it isn't the RAIDs fault that you gave a bad command, but it always seems if you care about your data, its worth the additional cost.

Peace,
JimBass