mdadm array troubles.

Gangrif · 09-04-2014, 09:00 AM

I have an intel storage array, 4-disk nfs/smb NAS job. A little while ago it failed, and I started down the path of trying to recover my data.

It turns out that under the covers, the thing is linux, and uses mdadm to build the array.

So took three of the disks (one of them is actually dead) and mounted them in a machine with adequate sata ports, and booted it up off of a fedora live disc.

It see's the mdadm array, but i cant get mdadm to assemble the array. It keeps telling me that a minimum of 3 disks are required, and that it can only detect 2.

using mdadm --examine on the individual disks gives me a hint that things are out of whack, and i'm wondering how to resolve them. I'll paste output from the live image in a bit, i'm posting from a different machine.

Basically, each disk tells me that a different number of drives are available, and i'm wondering if I can override that. Essentially, i want to fail the bad disk which isnt even present, and then make sure that the three working disks are active.

Gangrif · 09-04-2014, 09:08 AM

Ok, here's some output.

First, attempting to assemble the array based on uuid.

Code:

[root@localhost liveuser]# mdadm --assemble  --scan --uuid "dbeab1fc:fbec2bad:b0a43543:72ffeee2"
mdadm: /dev/md/1 assembled from 2 drives - not enough to start the array.
mdadm: No arrays found in config file or automatically

Now trying to specify the drives to use.

Code:

[root@localhost ~]# mdadm -v --assemble /dev/md1 /dev/sda3 /dev/sdb3 /dev/sdd3
mdadm: looking for devices for /dev/md1
mdadm: /dev/sda3 is identified as a member of /dev/md1, slot 1.
mdadm: /dev/sdb3 is identified as a member of /dev/md1, slot 0.
mdadm: /dev/sdd3 is identified as a member of /dev/md1, slot 2.
mdadm: added /dev/sda3 to /dev/md1 as 1 (possibly out of date)
mdadm: added /dev/sdd3 to /dev/md1 as 2
mdadm: no uptodate device for slot 6 of /dev/md1
mdadm: added /dev/sdb3 to /dev/md1 as 0
mdadm: /dev/md1 assembled from 2 drives - not enough to start the array.

Now we examine all of the available drives.

Code:

[root@localhost liveuser]# for i in a b d;do mdadm --examine /dev/sd${i}3;done;
/dev/sda3:
          Magic : a92b4efc
        Version : 0.90.01
           UUID : dbeab1fc:fbec2bad:b0a43543:72ffeee2
  Creation Time : Thu Aug  8 20:36:44 2013
     Raid Level : raid5
  Used Dev Size : 487991872 (465.39 GiB 499.70 GB)
     Array Size : 1463975616 (1396.16 GiB 1499.11 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 1

    Update Time : Sun Jul 20 18:17:04 2014
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 678fc865 - correct
         Events : 59722

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     1       8       19        1      active sync   /dev/sdb3

   0     0       8        3        0      active sync   /dev/sda3
   1     1       8       19        1      active sync   /dev/sdb3
   2     2       8       35        2      active sync
   3     3       8       51        3      active sync   /dev/sdd3
/dev/sdb3:
          Magic : a92b4efc
        Version : 0.90.01
           UUID : dbeab1fc:fbec2bad:b0a43543:72ffeee2
  Creation Time : Thu Aug  8 20:36:44 2013
     Raid Level : raid5
  Used Dev Size : 487991872 (465.39 GiB 499.70 GB)
     Array Size : 1463975616 (1396.16 GiB 1499.11 GB)
   Raid Devices : 4
  Total Devices : 3
Preferred Minor : 1

    Update Time : Tue Aug 26 22:40:13 2014
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 2
  Spare Devices : 0
       Checksum : 67c0cd55 - correct
         Events : 59724

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     0       8        3        0      active sync   /dev/sda3

   0     0       8        3        0      active sync   /dev/sda3
   1     1       0        0        1      faulty removed
   2     2       8       35        2      active sync
   3     3       0        0        3      faulty removed
/dev/sdd3:
          Magic : a92b4efc
        Version : 0.90.01
           UUID : dbeab1fc:fbec2bad:b0a43543:72ffeee2
  Creation Time : Thu Aug  8 20:36:44 2013
     Raid Level : raid5
  Used Dev Size : 487991872 (465.39 GiB 499.70 GB)
     Array Size : 1463975616 (1396.16 GiB 1499.11 GB)
   Raid Devices : 4
  Total Devices : 3
Preferred Minor : 1

    Update Time : Tue Aug 26 22:40:13 2014
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 2
  Spare Devices : 0
       Checksum : 67c0cd79 - correct
         Events : 59724

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     2       8       35        2      active sync

   0     0       8        3        0      active sync   /dev/sda3
   1     1       0        0        1      faulty removed
   2     2       8       35        2      active sync
   3     3       0        0        3      faulty removed

What concerns me is the "active devices" for sdb and sdd, they both claim 2. Also that block at the bottom for each drive, stating devices. b and d both show only sda, while a shows all 3.

Gangrif · 09-04-2014, 09:43 AM

I notice that the checksum is different on sda3. Makes me wonder if that disk was somehow not a member of the array with the 4th disk failed.

Which makes me think my data is likely lost.

Any information would be useful.

Thanks!