LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 01-27-2010, 10:02 AM   #1
circle69
LQ Newbie
 
Registered: Jan 2010
Location: Spain
Distribution: CentOS
Posts: 3

Rep: Reputation: 0
Unhappy Recover raid 5 array + LVM2


I explain what hapened last day. I had a raid 5 + lvm 2 array and lost a disk. While it was recovering the array, the power was down and recovery stopped.

When I recovered the power and start the machine the array was unable to start, it was degraded and the states were different between disks. Every disk watched the array in a different way. I put you the states:

Quote:

/dev/sdd1

Number Major Minor RaidDevice State
this 0 8 33 0 active sync /dev/sdc1

0 0 8 33 0 active sync /dev/sdc1
1 1 0 0 1 faulty removed
2 2 8 65 2 active sync /dev/sde1
3 3 8 81 3 active sync /dev/sdf1


/dev/sdd1:

Number Major Minor RaidDevice State
this 4 8 49 4 spare /dev/sdd1

0 0 0 0 0 removed
1 1 0 0 1 faulty removed
2 2 8 65 2 active sync /dev/sde1
3 3 8 81 3 active sync /dev/sdf1
4 4 8 49 4 spare /dev/sdd1

/dev/sde1

Number Major Minor RaidDevice State
this 2 8 65 2 active sync /dev/sde1

0 0 0 0 0 removed
1 1 0 0 1 faulty removed
2 2 8 65 2 active sync /dev/sde1
3 3 8 81 3 active sync /dev/sdf1
4 4 8 49 4 spare /dev/sdd1

/dev/sdf1

Number Major Minor RaidDevice State
this 3 8 81 3 active sync /dev/sdf1

0 0 0 0 0 removed
1 1 0 0 1 faulty removed
2 2 8 65 2 active sync /dev/sde1
3 3 8 81 3 active sync /dev/sdf1
4 4 8 49 4 spare /dev/sdd1
OK, reading something about that in the forum, I saw some people tried to do "mdadm --assemble --force /dev/mdX /dev/sdX[1-n]".
So I did that and it started to recover, it took abaout 9 hours until it finished but the array didn't started of course...

I post you the new raid devices states:

Quote:
mdadm --examine /dev/sdc1

/dev/sdc1:
Magic : a92b4efc
Version : 0.90.00
UUID : adf3ba9a:4ff9ca0b:faf240cc:71abd75b
Creation Time : Tue Nov 3 19:43:20 2009
Raid Level : raid5
Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
Array Size : 2930279808 (2794.53 GiB 3000.61 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 2

Update Time : Tue Jan 26 05:57:49 2010
State : clean
Active Devices : 2
Working Devices : 3
Failed Devices : 2
Spare Devices : 1
Checksum : e44e2f10 - correct
Events : 463462

Layout : left-symmetric
Chunk Size : 128K

Number Major Minor RaidDevice State
this 0 8 33 0 active sync /dev/sdc1

0 0 8 33 0 active sync /dev/sdc1
1 1 0 0 1 faulty removed
2 2 0 0 2 faulty removed
3 3 8 81 3 active sync /dev/sdf1
4 4 8 49 4 spare /dev/sdd1

/dev/sdd1:

Number Major Minor RaidDevice State
this 4 8 49 4 spare /dev/sdd1

0 0 8 33 0 active sync /dev/sdc1
1 1 0 0 1 faulty removed
2 2 0 0 2 faulty removed
3 3 8 81 3 active sync /dev/sdf1
4 4 8 49 4 spare /dev/sdd1

/dev/sde1

Number Major Minor RaidDevice State
this 2 8 65 2 active sync /dev/sde1

0 0 0 0 0 removed
1 1 0 0 1 faulty removed
2 2 8 65 2 active sync /dev/sde1
3 3 8 81 3 active sync /dev/sdf1
4 4 8 49 4 spare /dev/sdd1

/dev/sdf1

Number Major Minor RaidDevice State
this 3 8 81 3 active sync /dev/sdf1

0 0 8 33 0 active sync /dev/sdc1
1 1 0 0 1 faulty removed
2 2 0 0 2 faulty removed
3 3 8 81 3 active sync /dev/sdf1
4 4 8 49 4 spare /dev/sdd1
The first part in /dev/sdc1 is the same for all the devices, I just post you the states.
Another thing is tha all the devices say that there is no superblock

It seems that 3 disks are "active sync" but the states of the others doesn't match between them. And /dev/sdd1 is spare, the disk I added manually at first to start the recovery process.

Can I do something to recover the array? Have I lost my data?
I am totally lost and I don't know what to do next.

Thanks all and I am waitng for your answers.
 
Old 01-27-2010, 04:17 PM   #2
darvil
LQ Newbie
 
Registered: Jan 2010
Distribution: centOS
Posts: 19

Rep: Reputation: 1
Have you tried recovering the superblock? I've done it once or twice with similar issues but I can't remember all the steps I did. I used knoppix to fix the problem.
 
Old 01-28-2010, 03:51 AM   #3
circle69
LQ Newbie
 
Registered: Jan 2010
Location: Spain
Distribution: CentOS
Posts: 3

Original Poster
Rep: Reputation: 0
Thanks for your answer, I stopped doing tests over the array because I don't want to break it definitely.
If I try to recover the superblock, will it ruin the array or the data?

Thanks again.
 
Old 02-19-2010, 05:00 PM   #4
circle69
LQ Newbie
 
Registered: Jan 2010
Location: Spain
Distribution: CentOS
Posts: 3

Original Poster
Rep: Reputation: 0
Thumbs up

Well after a month of the disaster, finally I manage to recover the array.
I put the link where it is explained how to do it and the most important, the situation the things happened. It is very important to know what happened and the order of them.

Just to sum up, I lost 2 disks from a 4 disk array. First I lost one due a power failure and while it was recovering another started to fail with i/o errors.

http://kevin.deldycke.com/2008/07/he...data-recovery/

I hope it will help somebody else.
 
  


Reply

Tags
lvm2, mdadm, raid5


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Recover LVM2 with Data? bodaduk Linux - General 5 09-27-2010 01:16 PM
How to recover/mount RAID 0 array, dmraid? Robutux Linux - Hardware 3 07-20-2009 02:47 PM
proper way to use testdisk to recover missing logical volumes on a raid 5 w/ LVM2 markley268 Ubuntu 1 06-21-2009 01:02 PM
Trying to recover/mount RAID array sxa Linux - General 5 06-10-2009 10:32 AM
Using testdisk to recover deleted lvm2 partition sstendal Linux - General 2 04-02-2008 10:30 AM


All times are GMT -5. The time now is 09:46 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration