[SOLVED] How to set becomed spare to active in softraid

pbdiwpwm · 11-01-2011, 05:49 PM

Hi there,

I got an controller problem with my raid-6 disks.
After some reboots and much more tries to resync my array now I have a situation that only the last 2 disks are available with the same Events. One has 0 and the other has much less than the 1593222 from the array itself.
The latest two disk before I lost my active raid set are sdb3 and sdd3. The problem was that I have only could assembled the array back with only the drive /dev/sdb3. Then I tried to readd the disk /dev/sdd3 which becomes only a spare drive :-(.

Code:

~ # mdadm --examine /dev/sd[bd]3
/dev/sdb3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 45d7a2d4:628276ee:28b1e996:c44bc3c6
           Name : xen:datamd
  Creation Time : Sat Mar  5 01:23:00 2011
     Raid Level : raid6
   Raid Devices : 4

 Avail Dev Size : 488278016 (232.83 GiB 250.00 GB)
     Array Size : 976553984 (465.66 GiB 500.00 GB)
  Used Dev Size : 488276992 (232.83 GiB 250.00 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : b5f9456b:6b379c3f:0837a437:044b2aa8

Internal Bitmap : 2 sectors from superblock
    Update Time : Sun Oct 30 00:28:49 2011
       Checksum : c6ed17dd - correct
         Events : 1593222

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : A... ('A' == active, '.' == missing)
/dev/sdd3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 45d7a2d4:628276ee:28b1e996:c44bc3c6
           Name : xen:datamd
  Creation Time : Sat Mar  5 01:23:00 2011
     Raid Level : raid6
   Raid Devices : 4

 Avail Dev Size : 488278016 (232.83 GiB 250.00 GB)
     Array Size : 976553984 (465.66 GiB 500.00 GB)
  Used Dev Size : 488276992 (232.83 GiB 250.00 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 7417dcbe:6e6a90a2:f57919e5:74ec6027

Internal Bitmap : 2 sectors from superblock
    Update Time : Sun Oct 30 00:28:49 2011
       Checksum : aa234d01 - correct
         Events : 1593222

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : spare
   Array State : A... ('A' == active, '.' == missing)

Code:

~ # mdadm --misc --detail /dev/md125
/dev/md125:
        Version : 1.2
  Creation Time : Sat Mar  5 01:23:00 2011
     Raid Level : raid6
  Used Dev Size : 244138496 (232.83 GiB 250.00 GB)
   Raid Devices : 4
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Sun Oct 30 00:28:49 2011
          State : active, FAILED, Not Started
 Active Devices : 1
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 512K

           Name : xen:datamd
           UUID : 45d7a2d4:628276ee:28b1e996:c44bc3c6
         Events : 1593222

    Number   Major   Minor   RaidDevice State
       0       8       19        0      active sync   /dev/sdb3
       1       0        0        1      removed
       2       0        0        2      removed
       3       0        0        3      removed

       4       8       51        -      spare   /dev/sdd3

Code:

~ # mdadm --examine-bitmap /dev/sd[bd]3
        Filename : /dev/sdb3
           Magic : 6d746962
         Version : 4
            UUID : 45d7a2d4:628276ee:28b1e996:c44bc3c6
          Events : 1593222
  Events Cleared : 634730
           State : OK
       Chunksize : 64 MB
          Daemon : 5s flush period
      Write Mode : Normal
       Sync Size : 244138496 (232.83 GiB 250.00 GB)
          Bitmap : 3726 bits (chunks), 739 dirty (19.8%)
        Filename : /dev/sdd3
           Magic : 6d746962
         Version : 4
            UUID : 45d7a2d4:628276ee:28b1e996:c44bc3c6
          Events : 1593218
  Events Cleared : 634730
           State : OK
       Chunksize : 64 MB
          Daemon : 5s flush period
      Write Mode : Normal
       Sync Size : 244138496 (232.83 GiB 250.00 GB)
          Bitmap : 3726 bits (chunks), 739 dirty (19.8%)

Any idea to get the disk back as active one so the array could been restarted? I've no idea which slot the disk has to been had before.

cheers

pbdiwpwm · 11-01-2011, 07:52 PM

After some reads I found a solution by try and error.
The raid device contains an encrypted Luks device.
So I tried to used the spared disk in all disk slots from 3 to 2 to 1 by
assembling, than modifying the /sysfs and start the array. The check was simple:
as long I used the wrong slot cryptsetup does'nt find an luks-Device.

Code:

~ # mdadm --stop /dev/md125
mdadm: stopped /dev/md125
~ # mdadm --assemble --force --no-degraded /dev/md125 /dev/sdb3 /dev/sdd3
mdadm: /dev/md125 assembled from 1 drive (out of 4), but not started.
~ # cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md125 : inactive sdb3[0](S) sdd3[4](S)
      488278016 blocks super 1.2

md126 : active raid1 sda2[5](S) sdc2[4] sdb2[0] sdd2[3]
      19529656 blocks super 1.2 [3/3] [UUU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md127 : active raid1 sda1[5](S) sdc1[4] sdb1[0] sdd1[3]
      88345 blocks super 1.2 [3/3] [UUU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

unused devices: <none>
~ # mdadm --run --force /dev/md125
mdadm: failed to run array /dev/md125: Input/output error
~ # echo "2" >sys/block/md125/md/dev-sdd3/slot
~ # echo "in_sync" >/sys/block/md125/md/dev-sdd3/state
~ # cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md125 : inactive sdb3[0] sdd3[4]
      488278016 blocks super 1.2

md126 : active raid1 sda2[5](S) sdc2[4] sdb2[0] sdd2[3]
      19529656 blocks super 1.2 [3/3] [UUU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md127 : active raid1 sda1[5](S) sdc1[4] sdb1[0] sdd1[3]
      88345 blocks super 1.2 [3/3] [UUU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

unused devices: <none>
~ # mdadm --run /dev/md125
mdadm: started /dev/md125
~ # cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md125 : active raid6 sdb3[0] sdd3[4]
      488276992 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/2] [U_U_]
      bitmap: 2/2 pages [8KB], 65536KB chunk

md126 : active raid1 sda2[5](S) sdc2[4] sdb2[0] sdd2[3]
      19529656 blocks super 1.2 [3/3] [UUU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md127 : active raid1 sda1[5](S) sdc1[4] sdb1[0] sdd1[3]
      88345 blocks super 1.2 [3/3] [UUU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

unused devices: <none>
~ # cryptsetup luksOpen /dev/md125 md125_crypt
Device /dev/md125 is not a valid LUKS device.
~ # mdadm --stop /dev/md125
mdadm: stopped /dev/md125
~ # mdadm -A --force --no-degraded /dev/md125 /dev/sd[bd]3
mdadm: /dev/md125 assembled from 1 drive (out of 4), but not started.
~~ # cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md125 : inactive sdb3[0](S) sdd3[4](S)
      488278016 blocks super 1.2

md126 : active raid1 sda2[5](S) sdc2[4] sdb2[0] sdd2[3]
      19529656 blocks super 1.2 [3/3] [UUU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md127 : active raid1 sda1[5](S) sdc1[4] sdb1[0] sdd1[3]
      88345 blocks super 1.2 [3/3] [UUU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

unused devices: <none>
~ # mdadm --run --force /dev/md125
mdadm: failed to run array /dev/md125: Input/output error
~ # echo "in_sync" >/sys/block/md125/md/dev-sdd3/state
~ # echo "1" >/sys/block/md125/md/dev-sdd3/slot
~ # mdadm -Q --detail /dev/md125
/dev/md125:
        Version : 1.2
  Creation Time : Sat Mar  5 01:23:00 2011
     Raid Level : raid6
  Used Dev Size : 244138496 (232.83 GiB 250.00 GB)
   Raid Devices : 4
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Sun Oct 30 00:28:49 2011
          State : active, degraded, Not Started
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : xen:datamd
           UUID : 45d7a2d4:628276ee:28b1e996:c44bc3c6
         Events : 1593222

    Number   Major   Minor   RaidDevice State
       0       8       19        0      active sync   /dev/sdb3
       4       8       51        1      active sync   /dev/sdd3
       2       0        0        2      removed
       3       0        0        3      removed
~ # cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md125 : inactive sdb3[0] sdd3[4]
      488278016 blocks super 1.2

md126 : active raid1 sda2[5](S) sdc2[4] sdb2[0] sdd2[3]
      19529656 blocks super 1.2 [3/3] [UUU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md127 : active raid1 sda1[5](S) sdc1[4] sdb1[0] sdd1[3]
      88345 blocks super 1.2 [3/3] [UUU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

unused devices: <none>
~ # mdadm --run /dev/md125
mdadm: started /dev/md125
~ # cryptsetup luksOpen /dev/md125 md125_crypt
Enter passphrase for /dev/md125:

So I find the slot by trial and error. Pw request was ok and the
lvm could been found all my volumes in the partition.

I checked them all with fsck and it looks good for now.

Hopefully there is no other problem on data.

But in fact, the onboard controller on my server board from tyan (nv)
looks like sucks. I've now inserted an third one.