Recover raid 5 array + LVM2

circle69 · 01-27-2010, 09:02 AM

I explain what hapened last day. I had a raid 5 + lvm 2 array and lost a disk. While it was recovering the array, the power was down and recovery stopped.

When I recovered the power and start the machine the array was unable to start, it was degraded and the states were different between disks. Every disk watched the array in a different way. I put you the states:

Quote:

/dev/sdd1

Number Major Minor RaidDevice State
this 0 8 33 0 active sync /dev/sdc1

0 0 8 33 0 active sync /dev/sdc1
1 1 0 0 1 faulty removed
2 2 8 65 2 active sync /dev/sde1
3 3 8 81 3 active sync /dev/sdf1

/dev/sdd1:

Number Major Minor RaidDevice State
this 4 8 49 4 spare /dev/sdd1

0 0 0 0 0 removed
1 1 0 0 1 faulty removed
2 2 8 65 2 active sync /dev/sde1
3 3 8 81 3 active sync /dev/sdf1
4 4 8 49 4 spare /dev/sdd1

/dev/sde1

Number Major Minor RaidDevice State
this 2 8 65 2 active sync /dev/sde1

0 0 0 0 0 removed
1 1 0 0 1 faulty removed
2 2 8 65 2 active sync /dev/sde1
3 3 8 81 3 active sync /dev/sdf1
4 4 8 49 4 spare /dev/sdd1

/dev/sdf1

Number Major Minor RaidDevice State
this 3 8 81 3 active sync /dev/sdf1

0 0 0 0 0 removed
1 1 0 0 1 faulty removed
2 2 8 65 2 active sync /dev/sde1
3 3 8 81 3 active sync /dev/sdf1
4 4 8 49 4 spare /dev/sdd1

OK, reading something about that in the forum, I saw some people tried to do "mdadm --assemble --force /dev/mdX /dev/sdX[1-n]".
So I did that and it started to recover, it took abaout 9 hours until it finished but the array didn't started of course...

I post you the new raid devices states:

Quote:

mdadm --examine /dev/sdc1

/dev/sdc1:
Magic : a92b4efc
Version : 0.90.00
UUID : adf3ba9a:4ff9ca0b:faf240cc:71abd75b
Creation Time : Tue Nov 3 19:43:20 2009
Raid Level : raid5
Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
Array Size : 2930279808 (2794.53 GiB 3000.61 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 2

Update Time : Tue Jan 26 05:57:49 2010
State : clean
Active Devices : 2
Working Devices : 3
Failed Devices : 2
Spare Devices : 1
Checksum : e44e2f10 - correct
Events : 463462

Layout : left-symmetric
Chunk Size : 128K

Number Major Minor RaidDevice State
this 0 8 33 0 active sync /dev/sdc1

0 0 8 33 0 active sync /dev/sdc1
1 1 0 0 1 faulty removed
2 2 0 0 2 faulty removed
3 3 8 81 3 active sync /dev/sdf1
4 4 8 49 4 spare /dev/sdd1

/dev/sdd1:

Number Major Minor RaidDevice State
this 4 8 49 4 spare /dev/sdd1

0 0 8 33 0 active sync /dev/sdc1
1 1 0 0 1 faulty removed
2 2 0 0 2 faulty removed
3 3 8 81 3 active sync /dev/sdf1
4 4 8 49 4 spare /dev/sdd1

/dev/sde1

Number Major Minor RaidDevice State
this 2 8 65 2 active sync /dev/sde1

0 0 0 0 0 removed
1 1 0 0 1 faulty removed
2 2 8 65 2 active sync /dev/sde1
3 3 8 81 3 active sync /dev/sdf1
4 4 8 49 4 spare /dev/sdd1

/dev/sdf1

Number Major Minor RaidDevice State
this 3 8 81 3 active sync /dev/sdf1

0 0 8 33 0 active sync /dev/sdc1
1 1 0 0 1 faulty removed
2 2 0 0 2 faulty removed
3 3 8 81 3 active sync /dev/sdf1
4 4 8 49 4 spare /dev/sdd1

The first part in /dev/sdc1 is the same for all the devices, I just post you the states.
Another thing is tha all the devices say that there is no superblock

It seems that 3 disks are "active sync" but the states of the others doesn't match between them. And /dev/sdd1 is spare, the disk I added manually at first to start the recovery process.

Can I do something to recover the array? Have I lost my data?
I am totally lost and I don't know what to do next.

Thanks all and I am waitng for your answers.

darvil · 01-27-2010, 03:17 PM

Have you tried recovering the superblock? I've done it once or twice with similar issues but I can't remember all the steps I did. I used knoppix to fix the problem.

circle69 · 01-28-2010, 02:51 AM

Thanks for your answer, I stopped doing tests over the array because I don't want to break it definitely.
If I try to recover the superblock, will it ruin the array or the data?

Thanks again.

circle69 · 02-19-2010, 04:00 PM

Well after a month of the disaster, finally I manage to recover the array.
I put the link where it is explained how to do it and the most important, the situation the things happened. It is very important to know what happened and the order of them.

Just to sum up, I lost 2 disks from a 4 disk array. First I lost one due a power failure and while it was recovering another started to fail with i/o errors.

http://kevin.deldycke.com/2008/07/he...data-recovery/

I hope it will help somebody else.