mdadm: No partition table after recreation of existing array

Jside · 08-05-2015, 07:14 PM

So I had a system crash which resulted in sdd being marked as broken, and in the process of resyncing the disk, he machine crashed again which resulted in:

Code:

/dev/sdb:
         Events : 72592
/dev/sdc:
         Events : 72592
/dev/sdd:
         Events : 68456

Code:

/dev/sdb:
   Device Role : spare
/dev/sdc:
/dev/sdd:

I did run SMART self tests on all three which passed.

mdadm --assemble --force didn't worked because sdb is a spare and sdd is out of date, so I tried to manually recreate the array using:

Code:

mdadm --create --assume-clean --level=5 --raid-devices=3 --chunk=512K /dev/md127 /dev/sdb /dev/sdc missing

Which of course worked, but there is no partition table on it now, I had a partition under (md127p1) with a filesystem (I think it was either XFS or EXT4)

I haven't touched sdd so far. What have I done wrong? I searched a bit around the Internet and the exact same steps have worked for multiple persons. Have I gotten on offset or something wrong?

Here is the full mdadm --examine output before I did the recreation:

Code:

/dev/sdb:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x8
     Array UUID : 1c6d7aef:d82e2ee8:0ca9a5e4:95039a2e
           Name : <host>:0
  Creation Time : Sat Sep 22 14:27:56 2012
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
     Array Size : 3907025920 (3726.03 GiB 4000.79 GB)
  Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1960 sectors, after=1200 sectors
          State : clean
    Device UUID : a4f20363:44a6251f:329c0f30:946543de

    Update Time : Wed Aug  5 23:56:02 2015
  Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
       Checksum : 1e4b080d - correct
         Events : 72592

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : spare
   Array State : .A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 1c6d7aef:d82e2ee8:0ca9a5e4:95039a2e
           Name : <host>:0
  Creation Time : Sat Sep 22 14:27:56 2012
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
     Array Size : 3907025920 (3726.03 GiB 4000.79 GB)
  Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=1200 sectors
          State : clean
    Device UUID : 935a902e:8d106d19:f242cd13:e9a77bbe

    Update Time : Wed Aug  5 23:56:02 2015
       Checksum : a80cc307 - correct
         Events : 72592

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : .A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 1c6d7aef:d82e2ee8:0ca9a5e4:95039a2e
           Name : <host>:0
  Creation Time : Sat Sep 22 14:27:56 2012
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
     Array Size : 3907025920 (3726.03 GiB 4000.79 GB)
  Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=1200 sectors
          State : active
    Device UUID : 9b35e3bd:4e70109c:7000837a:b0ba7042

    Update Time : Wed Aug  5 23:52:56 2015
       Checksum : a4b0bd35 - correct
         Events : 68456

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)

And here is the mdadm --detail of the recreated array:

Code:

/dev/md127:
        Version : 1.2
  Creation Time : Thu Aug  6 01:28:15 2015
     Raid Level : raid5
     Array Size : 3907024896 (3726.03 GiB 4000.79 GB)
  Used Dev Size : 1953512448 (1863.01 GiB 2000.40 GB)
   Raid Devices : 3
  Total Devices : 2
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Thu Aug  6 01:28:15 2015
          State : clean, degraded 
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : <host>:127  (local to host <host>)
           UUID : 92efa2fc:3b3cca0b:dbd0090a:14a583f3
         Events : 0

    Number   Major   Minor   RaidDevice State
       0       8       48        0      active sync   /dev/sdd
       1       8       32        1      active sync   /dev/sdc
       4       0        0        4      removed

smallpond · 08-07-2015, 09:13 AM

It is unusual to do raid on the whole disks, then partition the raid. Usually, one creates partitions on each disk and then creates raid of the partitions.

My guess would be that you recreated the raid with the disks in a different order, which would lose access to all of the data. Create with assume-clean only writes the superblocks, so you could still recover from this if you have not written to the disks and can determine the correct order. Check for partition table with:

Code:

fdisk -l /dev/md127

S.Haran · 08-07-2015, 05:42 PM

Your "Array Sizes" do not match. I expect that is the source of your problem. I see this often with other cases. It's a common mistake.

Jside · 08-07-2015, 08:39 PM

I've managed to get it to work, the data offset was wrong (as you can see in the original --examine the disks had a data offset of 2048 sectors whereas the computed offset was not even close to that) and also when re-creating the array I've typed the disks in a wrong order as described above.

Thanks!