LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 11-17-2007, 05:45 AM   #1
auroraglacialis
LQ Newbie
 
Registered: Oct 2007
Posts: 9

Rep: Reputation: 0
Question Last Try: RAID 6 with 7 drives lost 2 drives without hardware errors. Recovery?


I used to have a RAID 6 (created with mdadm) consisting of 7 HDDs (5+2).
One disk (#7) died and I removed it.
I figured that there are still (5+1) disks left, so one spare still - and continued working with it until I can buy a new HDD. But then disaster struck and took out another drive (#6), /proc/mdstat told me there are now only 5 drives, so I had no reserve left.

I checked the disk that gave up last (#6) with SMART etc and it came out ok, so I figured it was maybe a software hickup and added it back in the RAID as a new drive (mdadm --add) and it started syncing. At about 2% the PC froze, upon reboot only 3 drives where in the RAID (#1,#2,#3).
I checked the missing drives (#4,#5) with mdadm -E and they where ok, superblocks and all. So I figured they where not in the RAID due to the crash before and I have to manually add them to the RAID for some reason.

Now the big mistake was to use --add for the first device (#5) I tried to get back in the RAID and looking at the RAID info, it was added as "spare".
Then strange things happened again with the PC and I checked for Hardware errors. Found that 2 IDE controllers where not behaving well any longer, probably causing all the trouble.

Now to get the data back I tried to copy every HD that was at one time part of the RAID to image files (with dd if=/dev/hdx of=/mnt/backup/hdx), so I would only have to use the on-board controller. So now I have 6 files which are images of the 6 HDDs that where formerly installed, 4 of which where pretty much untouched (#1,#2,#3,#4), one was added as a new drive that started syncing while the RAID was still active (#6) and one was added as spare while the RAID was inactive (#5).

Code:
"mdadm --assemble -f /dev/md0 /dev/loop1 /dev/loop2 /dev/loop3 /dev/loop4 /dev/loop5 /dev/loop6"
Trying to simply assemble the RAID from this fails with an IO-Error, using readonly gives no valid filesystem.

I figure that the data should well be there, since one drive was only 2% written on (the rest should still contain the old data from a time it was still in the RAID) and one drive was just added as spare (it was not written at all unless adding it as a spare deletes the contents). And I basically need only one of them to have 5 valid disks to start the RAID.

Now how can I recover at least some data?
Somehow telling the RAID to put the "spare" back in the place where it was before dropping out?
Recovering some data from the HDD that was in the RAID, dropped out and was added as a new HDD until it was about 2% in the resyncing process?

Hope someone can help me. I put quite some personal files on the RAID which are lost now.
I will try even desperate attempts since now I have about 2 Terrabyte of unused HDD space on the Disks that hold the old RAID data and I need that space soon... I can't affort a professional data recovery, so any suggestions are welcome at this point.

Many thanks
Aurora Glacialis

List of the Outputs of MDADM -E of all (backup) drives: (Important Note: The Number I gave above are not the actual numbers of the drives, so when I said "#6 was added", it does not mean that this drive is now "/dev/loop6" or "RaidDevice 6")

Code:
/dev/loop1:
          Magic : a92b4efc
        Version : 00.90.02
           UUID : a0f103b5:f0b078ea:1714136b:10b0e30d
  Creation Time : Tue Apr 25 22:19:59 2006
     Raid Level : raid6
    Device Size : 156288256 (149.05 GiB 160.04 GB)
     Array Size : 781441280 (745.24 GiB 800.20 GB)
   Raid Devices : 7
  Total Devices : 6
Preferred Minor : 0

    Update Time : Wed Jul 18 23:09:39 2007
          State : active
 Active Devices : 5
Working Devices : 6
 Failed Devices : 2
  Spare Devices : 1
       Checksum : f7ebc571 - correct
         Events : 0.18494993


      Number   Major   Minor   RaidDevice State
this     4      33        1        4      active sync

   0     0      22        1        0      active sync
   1     1      34       65        1      active sync
   2     2      56        1        2      active sync
   3     3       0        0        3      faulty removed
   4     4      33        1        4      active sync
   5     5       0        0        5      faulty removed
   6     6      57        1        6      active sync
   7     7      34        1        7      spare
/dev/loop2:
          Magic : a92b4efc
        Version : 00.90.02
           UUID : a0f103b5:f0b078ea:1714136b:10b0e30d
  Creation Time : Tue Apr 25 22:19:59 2006
     Raid Level : raid6
    Device Size : 156288256 (149.05 GiB 160.04 GB)
     Array Size : 781441280 (745.24 GiB 800.20 GB)
   Raid Devices : 7
  Total Devices : 6
Preferred Minor : 0

    Update Time : Wed Jul 18 23:09:39 2007
          State : active
 Active Devices : 5
Working Devices : 6
 Failed Devices : 2
  Spare Devices : 1
       Checksum : f7ebc58d - correct
         Events : 0.18494993


      Number   Major   Minor   RaidDevice State
this     6      57        1        6      active sync

   0     0      22        1        0      active sync
   1     1      34       65        1      active sync
   2     2      56        1        2      active sync
   3     3       0        0        3      faulty removed
   4     4      33        1        4      active sync
   5     5       0        0        5      faulty removed
   6     6      57        1        6      active sync
   7     7      34        1        7      spare
/dev/loop3:
          Magic : a92b4efc
        Version : 00.90.02
           UUID : a0f103b5:f0b078ea:1714136b:10b0e30d
  Creation Time : Tue Apr 25 22:19:59 2006
     Raid Level : raid6
    Device Size : 156288256 (149.05 GiB 160.04 GB)
     Array Size : 781441280 (745.24 GiB 800.20 GB)
   Raid Devices : 7
  Total Devices : 6
Preferred Minor : 0

    Update Time : Wed Jul 18 23:09:39 2007
          State : active
 Active Devices : 5
Working Devices : 6
 Failed Devices : 2
  Spare Devices : 1
       Checksum : f7ebc55e - correct
         Events : 0.18494993


      Number   Major   Minor   RaidDevice State
this     0      22        1        0      active sync

   0     0      22        1        0      active sync
   1     1      34       65        1      active sync
   2     2      56        1        2      active sync
   3     3       0        0        3      faulty removed
   4     4      33        1        4      active sync
   5     5       0        0        5      faulty removed
   6     6      57        1        6      active sync
   7     7      34        1        7      spare
/dev/loop4:
          Magic : a92b4efc
        Version : 00.90.02
           UUID : a0f103b5:f0b078ea:1714136b:10b0e30d
  Creation Time : Tue Apr 25 22:19:59 2006
     Raid Level : raid6
    Device Size : 156288256 (149.05 GiB 160.04 GB)
     Array Size : 781441280 (745.24 GiB 800.20 GB)
   Raid Devices : 7
  Total Devices : 6
Preferred Minor : 0

    Update Time : Wed Jul 18 23:09:39 2007
          State : active
 Active Devices : 5
Working Devices : 6
 Failed Devices : 2
  Spare Devices : 1
       Checksum : f7ebc584 - correct
         Events : 0.18494993


      Number   Major   Minor   RaidDevice State
this     2      56        1        2      active sync

   0     0      22        1        0      active sync
   1     1      34       65        1      active sync
   2     2      56        1        2      active sync
   3     3       0        0        3      faulty removed
   4     4      33        1        4      active sync
   5     5       0        0        5      faulty removed
   6     6      57        1        6      active sync
   7     7      34        1        7      spare
/dev/loop5:
          Magic : a92b4efc
        Version : 00.90.02
           UUID : a0f103b5:f0b078ea:1714136b:10b0e30d
  Creation Time : Tue Apr 25 22:19:59 2006
     Raid Level : raid6
    Device Size : 156288256 (149.05 GiB 160.04 GB)
     Array Size : 781441280 (745.24 GiB 800.20 GB)
   Raid Devices : 7
  Total Devices : 6
Preferred Minor : 0

    Update Time : Wed Jul 18 23:09:39 2007
          State : active
 Active Devices : 5
Working Devices : 6
 Failed Devices : 2
  Spare Devices : 1
       Checksum : f7ebc5ac - correct
         Events : 0.18494993


      Number   Major   Minor   RaidDevice State
this     1      34       65        1      active sync

   0     0      22        1        0      active sync
   1     1      34       65        1      active sync
   2     2      56        1        2      active sync
   3     3       0        0        3      faulty removed
   4     4      33        1        4      active sync
   5     5       0        0        5      faulty removed
   6     6      57        1        6      active sync
   7     7      34        1        7      spare
/dev/loop6:
          Magic : a92b4efc
        Version : 00.90.02
           UUID : a0f103b5:f0b078ea:1714136b:10b0e30d
  Creation Time : Tue Apr 25 22:19:59 2006
     Raid Level : raid6
    Device Size : 156288256 (149.05 GiB 160.04 GB)
     Array Size : 781441280 (745.24 GiB 800.20 GB)
   Raid Devices : 7
  Total Devices : 6
Preferred Minor : 0

    Update Time : Wed Jul 18 23:09:39 2007
          State : active
 Active Devices : 5
Working Devices : 6
 Failed Devices : 2
  Spare Devices : 1
       Checksum : f7ebc572 - correct
         Events : 0.18494993


      Number   Major   Minor   RaidDevice State
this     7      34        1        7      spare

   0     0      22        1        0      active sync
   1     1      34       65        1      active sync
   2     2      56        1        2      active sync
   3     3       0        0        3      faulty removed
   4     4      33        1        4      active sync
   5     5       0        0        5      faulty removed
   6     6      57        1        6      active sync
   7     7      34        1        7      spare
Code:
Personalities : [linear] [raid0] [raid1] [raid5] [raid4] [raid6] [multipath] [faulty] 
md0 : inactive loop3[0] loop6[7](S) loop2[6] loop1[4] loop4[2] loop5[1]
      937729536 blocks
       
unused devices: <none>
 
Old 12-12-2007, 08:35 AM   #2
auroraglacialis
LQ Newbie
 
Registered: Oct 2007
Posts: 9

Original Poster
Rep: Reputation: 0
Hi again.

So no hints at all? *sigh* That's sad, I was really hoping for at least some tips for desperate attempts. Anything but simply formatting the disks and forget all the data. However, I need the storage capacity soon, so I would appreciate even tips that have a high risk of failure - anything is better than doing nothing, right?

Greetings
Aurora
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
How to fix root ownership on recovery with existing drives drs_it Red Hat 1 07-30-2007 11:39 AM
I have lost my optical drives! Zedd129 Linux - Hardware 6 10-06-2006 07:22 PM
Adding additional drives/resizing drives in RAID arrays kheldar Linux - Hardware 1 10-19-2005 02:21 PM
Upgrading hard drives on Software raid 1 boot drives. linuxboy123 Linux - General 0 12-11-2003 04:28 PM
raid on 2 different drives ronss Linux - Hardware 2 08-24-2002 08:27 AM


All times are GMT -5. The time now is 11:01 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration