LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 04-04-2011, 04:48 AM   #1
nicolasdiogo
Member
 
Registered: Oct 2003
Location: UK
Distribution: debian lenny x64
Posts: 129

Rep: Reputation: 20
Red face RAID 10 failed - and my installation uses a lvm on top - how can i recover from this?


Hello,

i am using Debian on my system and had a problem over the weekend (not known)
today when trying to reboot it - i get an error message that goes explains that it can not mount md0

there are others messages that says:
md: kicking non-fresh sde2 from array
...
md: kicking non-fresh sdf2 from array
..

please see attachment for dmesg

i have already found this post here:
http://www.linuxquestions.org/questi...-array-416853/

but its suggestion does not work for me.

any idea how i can recover from this.

my system has md0 made up of 6 disks and has lvm on top of it.

i am really out of ideas here - please any suggestions? (note the desperation in my tone!)

thanks,

nicolas
Attached Files
File Type: txt myServer_dmesg.txt (13.8 KB, 5 views)

Last edited by nicolasdiogo; 04-04-2011 at 05:06 AM. Reason: added dmesg
 
Old 04-04-2011, 06:06 AM   #2
nicolasdiogo
Member
 
Registered: Oct 2003
Location: UK
Distribution: debian lenny x64
Posts: 129

Original Poster
Rep: Reputation: 20
i will try to baby-step into this problem.

since my root partition is in LVM and it depends on RAID10 - i can not mount this partition until i have fixed RAID10 to load LVM to then edit the necessary config files.

this is a catch 22..

i suppose i have to use a live distro to 'create' my RAID10 and LVM and then mount its root.

suggestions?
 
Old 04-04-2011, 11:50 AM   #3
never say never
Member
 
Registered: Sep 2009
Location: Indiana, USA
Distribution: SLES, SLED, OpenSuse, CentOS, ubuntu 10.10, OpenBSD, FreeBSD
Posts: 195

Rep: Reputation: 37
Gotta hate this first thing Monday morning.

This might be of use check out mdadm man pages.
Quote:
--assume-clean
Tell mdadm that the array pre-existed and is known to be clean. It can be useful when trying to recover from a major failure as you can be sure that no data will be affected unless you actually write to the array. It can also be used when creating a RAID1 or RAID10 if you want to avoid the initial resync, however this practice - while normally safe - is not recommended. Use this ony if you really know what you are doing.
However, I would only mount read only for data recovery, then rebuild the array, if it were me.

Also do you know what caused the 2 drives to fail? It appears (guessing here) that sde2 and sdf2 are a Raid1 set that is part of a larger Raid 0 Volume, is that correct?

When designing Raid 10 (or any Raid for that matter) always look at things like the power and SATA controllers for the drives. For Raid 10 set up each Raid 1 set so that each drive of the set is on a different SATA channel, and different power supply.

In your case, assuming your setup is like this:
sda3, sdb3 - Raid 1 (Set 1)
sdc2, sdd2 - Raid 1 (Set 2)
sde2, sdf2 - Raid 1 (Set 3)

With these three Raid 1 sets part of md0; I would make sure that
sda3, sdc2, sde2 are not connected to the same power or SATA controller as their mirror. This helps to limit the possibility of critical failure like you experienced.

Good luck, please let us know if / how you recover.
 
Old 04-16-2011, 05:45 AM   #4
nicolasdiogo
Member
 
Registered: Oct 2003
Location: UK
Distribution: debian lenny x64
Posts: 129

Original Poster
Rep: Reputation: 20
thanks for the suggestions

it was a complete disk disaster.

3 disks failed - after a power surge.
i have bought a new power protection. and i have contacted the company that produced my previous one as there was a 10000 warranty to any products attached to it. but it seems unlikely that i will get anything.

i had to recover from backups after replacing the 1 of the disks that was broken (just would not power up).

but i am sure that i could have recovered this installation - it must have been something that missed while being under pressure to do 'something' that i did something wrong.

could you please suggest me information or a link to which steps i should take to collect sufficient information to reassemble a RAID and LVM PRIOR to a disaster?


thanks,

Nicolas
 
Old 05-19-2011, 06:17 AM   #5
never say never
Member
 
Registered: Sep 2009
Location: Indiana, USA
Distribution: SLES, SLED, OpenSuse, CentOS, ubuntu 10.10, OpenBSD, FreeBSD
Posts: 195

Rep: Reputation: 37
My best advise is to simulate disasters on non-production systems. Build an array with the same or similar make up (perhaps using older / smaller drives), then simulate various failures. You will then have plenty of time to figure out how to recover, and document how you did it, from each failure. Then if (when) it happens on the production equipment you are prepared. Really no deifferent than restoring a tape backup so that you are familiar with the process before you have to do it under pressure.

Personally, I am in the process of moving all my servers over to VMWare ESXI (the free version). I am creating multiple datastores, backing up the data and images between the stores. In the event of failure, I have to manually move things around and restart servers, but I don't care (within reason) what that equipment is, or where the data is.
 
Old 05-20-2011, 12:22 PM   #6
trey85stang
Senior Member
 
Registered: Sep 2003
Posts: 1,090

Rep: Reputation: 41
Quote:
Originally Posted by never say never View Post
My best advise is to simulate disasters on non-production systems. Build an array with the same or similar make up (perhaps using older / smaller drives), then simulate various failures. You will then have plenty of time to figure out how to recover, and document how you did it, from each failure. Then if (when) it happens on the production equipment you are prepared. Really no deifferent than restoring a tape backup so that you are familiar with the process before you have to do it under pressure.

Personally, I am in the process of moving all my servers over to VMWare ESXI (the free version). I am creating multiple datastores, backing up the data and images between the stores. In the event of failure, I have to manually move things around and restart servers, but I don't care (within reason) what that equipment is, or where the data is.
good call, all my fancy software/hardware raid setups go through a complete DR tests with documentation before they go into production. Of course this is in a corporate environment but even if you are a home based user you should have a DR plan if you are doing some kind of fancy raiding... Especially if the data is important to you.

I've never worked with software raid 10, so Im not sure how to help... I understand you can lose 3 drives... with a raid10 but it would have to be specific drives. Raid 5 w/ 2 hotspares would of saved you though. Of course you never really plan on losing 3 disks in one shot.
 
  


Reply

Tags
debian, raid10, recover


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Recover Raid+LVM on non system drives after system failure spindley Linux - General 4 01-19-2011 04:18 PM
Opensuse 11 software RAID 5 failed: how to recover ? laufandreas Linux - Server 3 06-30-2009 04:51 AM
intel matrix raid - how to recover failed harddisk qqqson Linux - Hardware 0 03-27-2008 09:44 AM
howto recover LVM on RAID-6 data? Tahir Saleh Linux - Newbie 0 11-19-2007 06:19 AM
Should I use LVM on top of 3ware Hardware RAID? bfoddy Linux - Hardware 2 09-22-2007 04:06 PM


All times are GMT -5. The time now is 08:03 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration