Failed RAID array

daisychick · 11-27-2014, 11:51 PM

I have a machine where the sata card failed and caused me to lose my raid array. I tested the drives externally and they are in good shape. I have moved the drives to a new machine (with enough onboard sata ports) and am attempting to get the array back up but I've run into problems. It's a RAID 5 array with 4 drives. It appears one of the drives is not in the array. I'm running ubuntu 14.04 and the current mdadm.conf is still the default. I have the old mdadm.conf information. I'd really like to preserve the data on the drives. Suggestions?

Here's what mdadm has to say:

Code:

cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md127 : inactive sdd1[2] sde1[4] sdb1[5]
      5860537608 blocks super 1.2

Code:

sudo mdadm --examine /dev/sd[bcde]1
/dev/sdb1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x2
     Array UUID : ef1b89aa:a2bbaab9:2ce4bdd0:47832afb
           Name : mediaserver:0
  Creation Time : Tue Aug  6 08:38:10 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3907025072 (1863.01 GiB 2000.40 GB)
     Array Size : 5860535808 (5589.04 GiB 6001.19 GB)
  Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
Recovery Offset : 1799197544 sectors
   Unused Space : before=1968 sectors, after=1200 sectors
          State : clean
    Device UUID : 968168e5:6c695c7b:66a78ba1:9bccdc80

    Update Time : Sun Sep 29 16:09:54 2013
       Checksum : dd82d9a - correct
         Events : 47568

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : .AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : ef1b89aa:a2bbaab9:2ce4bdd0:47832afb
           Name : mediaserver:0
  Creation Time : Tue Aug  6 08:38:10 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3907025072 (1863.01 GiB 2000.40 GB)
     Array Size : 5860535808 (5589.04 GiB 6001.19 GB)
  Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=1200 sectors
          State : clean
    Device UUID : 1da9199e:cc6fc424:128e502c:881d890a

    Update Time : Sun Sep 29 14:46:26 2013
       Checksum : 1923ec18 - correct
         Events : 47565

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : ef1b89aa:a2bbaab9:2ce4bdd0:47832afb
           Name : mediaserver:0
  Creation Time : Tue Aug  6 08:38:10 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3907025072 (1863.01 GiB 2000.40 GB)
     Array Size : 5860535808 (5589.04 GiB 6001.19 GB)
  Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=1200 sectors
          State : clean
    Device UUID : 3688a662:e042319b:017fe47f:a73d04b2

    Update Time : Sun Sep 29 16:09:54 2013
       Checksum : 4f2dc2e7 - correct
         Events : 47568

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : .AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : ef1b89aa:a2bbaab9:2ce4bdd0:47832afb
           Name : mediaserver:0
  Creation Time : Tue Aug  6 08:38:10 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3907025072 (1863.01 GiB 2000.40 GB)
     Array Size : 5860535808 (5589.04 GiB 6001.19 GB)
  Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=1200 sectors
          State : clean
    Device UUID : 173cc321:8c8b416a:635388f7:3dd0bb3e

    Update Time : Sun Sep 29 16:09:54 2013
       Checksum : e1b6266d - correct
         Events : 47568

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : .AAA ('A' == active, '.' == missing, 'R' == replacing)

With array stopped

Code:

sudo mdadm --assemble --scan -v
mdadm: looking for devices for further assembly
mdadm: no RAID superblock on /dev/sde
mdadm: no RAID superblock on /dev/sdd
mdadm: no RAID superblock on /dev/sdc
mdadm: no RAID superblock on /dev/sdb
mdadm: no RAID superblock on /dev/sda5
mdadm: no RAID superblock on /dev/sda2
mdadm: no RAID superblock on /dev/sda1
mdadm: no RAID superblock on /dev/sda
mdadm: /dev/sde1 is identified as a member of /dev/md/mediaserver:0, slot 3.
mdadm: /dev/sdd1 is identified as a member of /dev/md/mediaserver:0, slot 1.
mdadm: /dev/sdc1 is identified as a member of /dev/md/mediaserver:0, slot 0.
mdadm: /dev/sdb1 is identified as a member of /dev/md/mediaserver:0, slot 2.
mdadm: added /dev/sdc1 to /dev/md/mediaserver:0 as 0 (possibly out of date)
mdadm: added /dev/sdb1 to /dev/md/mediaserver:0 as 2
mdadm: added /dev/sde1 to /dev/md/mediaserver:0 as 3
mdadm: added /dev/sdd1 to /dev/md/mediaserver:0 as 1
mdadm: /dev/md/mediaserver:0 assembled from 2 drives and 1 rebuilding - not enough to start the array.
mdadm: looking for devices for further assembly
mdadm: No arrays found in config file or automatically

Code:

sudo mdadm --detail --scan
ARRAY /dev/md127 metadata=1.2 spares=1 name=mediaserver:0 UUID=ef1b89aa:a2bbaab9:2ce4bdd0:47832afb

Soapm · 11-28-2014, 12:57 AM

Code:

 Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

C is the only drive that thinks everything is cool. Notice the other drives says one is missing.

I would think you can just re-add it and let the raid rebuild itself.

Code:

mdadm --manage /dev/md127 --re-add /dev/sdc1

daisychick · 11-28-2014, 01:22 AM

thanks for the quick response but no luck...

Code:

sudo mdadm --manage /dev/md127 --re-add /dev/sdc1
mdadm: --re-add for /dev/sdc1 to /dev/md127 is not possible

daisychick · 11-28-2014, 01:35 AM

I also tried:

Code:

sudo mdadm --assemble /dev/md127 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1
mdadm: /dev/md127 assembled from 2 drives and 1 rebuilding - not enough to start the array.

Soapm · 11-28-2014, 02:20 AM

I'm no expert so use my help at your own risk. I would first try and force a repair...

Code:

mdadm --assemble --scan --force

or

Code:

mdadm --assemble --run --force /dev/md127 /dev/sd[cde]1

If that doesn't work, make sure your other 3 drives are working correctly in the array, then you can zero the suerblock on the faulty drive and add it back like it's new. Make sure you have a functioning array before you try this or it will begin to rebuild and you will loose all your stuff.

Code:

mdadm --zero-superblock /dev/sdc1

Code:

mdadm --add /dev/md127 /dev/sdc1

The array should be working and mountable with 3 drives. If the array is working, and you have spare drives, you may want to back up all your data and start the array over from scratch. It shouldn't be necessary, but it will work as a last resort.

Soapm · 11-28-2014, 02:29 AM

Quote:

Originally Posted by daisychick

I also tried:

Code:

cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md127 : inactive sdd1[2] sde1[4] sdb1[5]
      5860537608 blocks super 1.2

Code:

sudo mdadm --assemble /dev/md127 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1
mdadm: /dev/md127 assembled from 2 drives and 1 rebuilding - not enough to start the array.

ps... These two results don't match, the former is saying you have 3 drives working in an array. The later says you have 2 working drives and the array is rebuilding, if that is the case then you may have lost all your stuff already...

Post the out put of this command...

Code:

mdadm --detail /dev/md127

daisychick · 11-28-2014, 04:01 AM

Here you go.

Code:

 sudo mdadm --detail /dev/md127
/dev/md127:
        Version : 1.2
  Creation Time : Tue Aug  6 08:38:10 2013
     Raid Level : raid5
  Used Dev Size : 1953511936 (1863.01 GiB 2000.40 GB)
   Raid Devices : 4
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Sun Sep 29 16:09:54 2013
          State : active, FAILED, Not Started
 Active Devices : 2
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 512K

           Name : mediaserver:0
           UUID : ef1b89aa:a2bbaab9:2ce4bdd0:47832afb
         Events : 47568

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       2       8       49        1      active sync   /dev/sdd1
       5       8       17        2      spare rebuilding   /dev/sdb1
       4       8       65        3      active sync   /dev/sde1

Soapm · 11-28-2014, 04:09 PM

I'm puzzled, sdc is removed which is why you can't re-add it but I would caution zeroing the superblock and adding it back since sdb says it's "rebuilding". I do know you can't have a raid 5 with only 2 active devices. If the force command using [bde] doesn't work then I'm afraid I wouldn't know how to safely proceed.

Code:

mdadm --assemble --force -vv /dev/md127 /dev/sdb1 /dev/sdd1 /dev/sde1

I do see that the feature map is different on sdb but I have no idea what that means.

Code:

Feature Map : 0x2

I also notice it has this???

Code:

Recovery Offset : 1799197544 sectors

Also, post the output of this command to see if it really is rebuilding and how far along it is; or is that error indicating sdb needs to be rebuilt but isn't actually doing so???

Code:

watch -n1 cat /proc/mdstat

ps.. I also noticed that sdc had a different last update time, looks like it was kicked out before the others on the failed sata card. And it could be that sdb was syncing when the sata card took its last breath...

Code:

Update Time : Sun Sep 29 14:46:26 2013

Here is a link that might help...

http://unix.stackexchange.com/questi...md-raid5-array

Down toward the bottom under answers it gives this advice;

Quote:

To conclude, I think that if the above command fails, you should try to recreate the array like this:

Code:

mdadm --create /dev/md1 --assume-clean -l5 -n4 -c64 /dev/sd[abc]2 missing

If you do the --create, the missing part is important, don't try to add a fourth drive in the array, because then construction will begin and you will lose your data. Creating the array with a missing drive, will not change its contents and you'll have the chance to get a copy elsewhere (raid5 doesn't work the same way as raid1).

If that fails to bring the array up, try this solution (perl script) here Recreating an array

If you finally manage to bring the array up, the filesystem will be unclean and probably corrupted. If one disk fails during rebuild, it is expected that the array will stop and freeze not doing any writes to the other disks. In this case two disks failed, maybe the system was performing write requests that wasn't able to complete, so there is some small chance you lost some data, but also a chance that you will never notice it :-)

edit: some clarification added.

https://raid.wiki.kernel.org/index.p...ating_an_array

daisychick · 11-29-2014, 12:55 AM

I can say it's not actually rebuilding. There's usually a progress update but it just continues to state it's inactive.

Code:

sudo mdadm --assemble --force -vv /dev/md127 /dev/sdb1 /dev/sdd1 /dev/sde1
mdadm: looking for devices for /dev/md127
mdadm: /dev/sdb1 is identified as a member of /dev/md127, slot 2.
mdadm: /dev/sdd1 is identified as a member of /dev/md127, slot 1.
mdadm: /dev/sde1 is identified as a member of /dev/md127, slot 3.
mdadm: no uptodate device for slot 0 of /dev/md127
mdadm: added /dev/sdb1 to /dev/md127 as 2
mdadm: added /dev/sde1 to /dev/md127 as 3
mdadm: added /dev/sdd1 to /dev/md127 as 1
mdadm: /dev/md127 assembled from 2 drives and 1 rebuilding - not enough to start the array.

Soapm · 11-29-2014, 10:26 PM

Just curious, did you try the create with the assume clean?

Ser Olmy · 11-29-2014, 11:05 PM

Could you post the original mdadm.conf as well as the output from mdadm --examine for each array member?

(I wouldn't recommend using --assume-clean unless you're absolutely sure all the RAID members are 100% OK. If one of the drives in the RAID 5 array are out of sync, the driver will use possibly out-of-date information from that member rather than regenerating the data using information from the parity block/strip.)

daisychick · 12-01-2014, 11:24 PM

Thanks so much for the tips! The --examine on all members of the array is in the first post. The old mdadm.conf is below. Thanks!

Code:

# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays
#ARRAY /dev/md0  UUID=ef1b89aa:a2bbaab9:2ce4bdd0:47832afb

# This file was auto-generated on Mon, 11 Jun 2012 18:04:27 -0500
# by mkconf $Id$

daisychick · 12-01-2014, 11:29 PM

Three different folks on three different sites suggested use --force so I did. Not sure that was the best idea in retrospect. Here's what I got...

Code:

sudo mdadm --assemble --force /dev/md127 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1
mdadm: forcing event count in /dev/sdc1(0) from 47565 upto 47568
mdadm: /dev/md127 assembled from 3 drives and 1 rebuilding - not enough to start the array.

edit: definitely think that was a bad idea... they're all marked spares now...

Code:

cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md127 : inactive sdd1[2](S) sdc1[0](S) sde1[4](S) sdb1[5](S)
      7814050144 blocks super 1.2

Soapm · 12-02-2014, 02:54 AM

I apologize if I caused any problems and I hope Ser Olmy has a solution because as I stated, you've gone beyond my comfort level of helping others but I will say that your last attempt to assemble included all 4 drives. It's probably good that didn't work since I believe it would have kicked off a rebuild that would have overwritten your data.

I believe what made your previous attempt with 3 drives fail was this entry in sdb...

Code:

Recovery Offset : 1799197544 sectors

Perhaps using what was [cde] would have been the ticket since --force overlooks "out of date" but not out of sync meta data??? You want a degraded array with 3 drives initially and not all 4.

I don't think the spare status is a setback, that will change when the array is brought up. The main thing is don't kick off a rebuild or you're toast.

This is from the man on --force...

Quote:

Assemble the array even if the metadata on some devices appears to be out-of-date. If mdadm cannot find enough working devices to start the array, but can find some devices that are recorded as having failed, then it will mark those devices as working so that the array can be started. An array which requires --force to be started may contain data corruption. Use it carefully.

That's why I thought [cde] might work since the problem with sdc, besides the card port failing, is that it's update time was off by two hours. If nothing was written to the array in those two hours then...

Perhaps this will put you a bit at ease and help you relax, I had two drives fail in a raid 5 at different times and someone helped me bring it back up. I was able to dd one of the failed drives onto a new drive, and they walked me through rewriting the superblock to alter the uuid and presto... They explained to me at that time, as long as the data is undisturbed (re-sync or rebuild) then your array is more than likely salvageable, it's just a matter of getting super blocks that make mdadm happy and the drives readable... I didn't know my chunk size so we tried different ones until the drives could be read. I'm not suggesting my experience but I'm trying to show you how resilient an array really is.

Good luck to you...

daisychick · 12-03-2014, 12:12 AM

That is some GREAT news! And I agree, I'm sure the data is there. I just need to find someone smart enough to help me get to it.

I keep this pc powered off unless someone has a new suggestion just in case.