I've had software RAID 5 arrays for a while now, so they were set up before a RAID array could be partitioned. I had two separate RAID 5 arrays on the same set of drives. One was for / and the other for /home.

I moved the / to an SSD and figured I'd expand the other RAID array by failing a drive, repartitioning it then adding it back in. After repeating for the remaining drives, I could then expand the RAID array to use the full size of the drives.

Partway through the second drive being added back in, the RAID array stopped with a kernel error. The drive I was adding and another drive both showed as failed.

I couldn't restart the array so I copied the failed drive (Seagate's SeaTools did show it as faulty, but without SMART being tripped) to a new one and tried again. dd_rescue reported the drive copied correctly but I still couldn't restart the array.

So I tried the old standby of recreating the array. This allows me to start it but the ext3 file system won't mount. So I then tried my script (listed in another thread) to try every combination of drives to assemble the array and mount the file system. Still no luck.

At this point I think I'm screwed, but does anyone have any ideas?

The issue was a misunderstanding of RAID5. RAID1 is mirroring so each disk is an exact duplicate of the others and your plan might have worked for a RAID1. RAID5 on the other hand stripes data and parity bits across all the disks. You can lose 1 parity bit or 1 data bit for each stripe and it will continue to run with the remaining data/parity bits but if you lose more than 1 in a given stripe it is hosed. It can rebuild the missing data or parity bit when you add a new drive because the information it still has tells it about the whole stripe.

It appears you lost two drives at the same time and that is always fatal for RAID5 because it means you've lost 2 bits (either 1 data and 1 parity or 2 data) for each stripe and therefore don't have enough information to recover the RAID5 set.

There is no confusion between the different RAID levels. The plan for expanding the array would have worked if the disk error hadn't bunged up the array. Once the underlying partitions are large enough, resizing the array to use the full space is trivial.

The problem was an apparently botched write to one drive by the operating system which unfortunately occurred while another drive was resyncing. In theory it should have just affected a small amount of disk space which might have been recoverable or at least tolerable.

However, it appears that it is preventing the file system from being recognized. That's why I was hoping that someone might have an idea on a way around the problem - to get a flaky file system back.

Anyway, I eventually decided to scrap the array and recover the files as best I can from other sources.