recover encrypted raid5

DeepB · 07-10-2014, 03:29 PM

Hallo,

I have (had

) a dm-crypt (luks) encrypted raid5.

This raid5 (with 3 drives) dropped 2 drives (not both at the same time). The array did however not go into readonly, so I have 3 drives each with a different number of events.

mdadm --examine:

Code:

/dev/sdd1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x8
     Array UUID : 9eb8d131:c620b82c:8a35244f:322f316e
           Name : dcerouter:9
  Creation Time : Sat Mar 31 16:16:22 2012
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 5860119439 (2794.32 GiB 3000.38 GB)
     Array Size : 5860118528 (5588.64 GiB 6000.76 GB)
  Used Dev Size : 5860118528 (2794.32 GiB 3000.38 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1960 sectors, after=911 sectors
          State : clean
    Device UUID : 9c33d9e0:698ed6de:e83d6c56:5a066145

    Update Time : Thu Jul 10 19:01:13 2014
  Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
       Checksum : 55b4f6a1 - correct
         Events : 313003

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : A.A ('A' == active, '.' == missing, 'R' == replacing)

Code:

/dev/sde1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 9eb8d131:c620b82c:8a35244f:322f316e
           Name : dcerouter:9
  Creation Time : Sat Mar 31 16:16:22 2012
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 5860119439 (2794.32 GiB 3000.38 GB)
     Array Size : 5860118528 (5588.64 GiB 6000.76 GB)
  Used Dev Size : 5860118528 (2794.32 GiB 3000.38 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=911 sectors
          State : clean
    Device UUID : 506d89c0:209526a3:a507611b:46cdebb1

    Update Time : Thu Jul 10 19:01:13 2014
       Checksum : 2b2cc764 - correct
         Events : 313003

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : A.A ('A' == active, '.' == missing, 'R' == replacing)

Code:

/dev/sdf1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 9eb8d131:c620b82c:8a35244f:322f316e
           Name : dcerouter:9
  Creation Time : Sat Mar 31 16:16:22 2012
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 5860119439 (2794.32 GiB 3000.38 GB)
     Array Size : 5860118528 (5588.64 GiB 6000.76 GB)
  Used Dev Size : 5860118528 (2794.32 GiB 3000.38 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=911 sectors
          State : active
    Device UUID : f86618cd:ddb957ed:80f2388e:9275433e

    Update Time : Mon Jun 30 22:15:00 2014
       Checksum : 81118dd0 - correct
         Events : 248091

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)

The events of two drives were quite close so I force-assembled the array with

Code:

 mdadm -A --force /dev/md9 /dev/sd[gf]1

my /proc/mdstat:

Code:

md9 : active (auto-read-only) raid5 sdd1[4] sde1[3]
      5860118528 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [U_U]

i can use

Code:

cryptsetup luksOpen /dev/md9 data

however I cannot mount data, it says

Code:

mount: you must specify the filesystem type

Fsck returns:

Code:

fsck from util-linux 2.20.1
e2fsck 1.42.10 (18-May-2014)
fsck.ext2: Attempt to read block from filesystem resulted in short read while trying to open /dev/mapper/data
Could this be a zero-length partition?

Any ideas what I could do to recover the data?

thank you
Daniel

S.Haran · 07-11-2014, 09:10 AM

Can you inspect /dev/mapper/data with a hexeditor?

Does it look unencrypted?

If so try testdisk and see if it can find a filesystem.

DeepB · 07-12-2014, 03:22 AM

if I open it with a hex editor (hte) it says

Code:

unhandled exception: I/O error: Input/output error

TestDisk returns:

Code:

Disk /dev/mapper/data - 6000 GB / 5588 GiB - 11720232960 sectors

The harddisk (6000 GB / 5588 GiB) seems too small! (< 14840 TB / 13497 TiB)
Check the harddisk size: HD jumpers settings, BIOS detection...

The following partitions can't be recovered:
     Partition               Start        End    Size in sectors
>  HPFS - NTFS           11482089323 28985325438826 28973843349504
   FAT16 <32M            11503427522 12395918144  892490623
   FAT12                 11592550862 14419265165 2826714304
   FAT32 LBA             11606904376 12783383285 1176478910
   FAT16 >32M            11627498189 12530370854  902872666
   FAT16 <32M            11640546771 12953765491 1313218721
   HPFS - NTFS           11644838697 15120180024 3475341328
   FAT12                 11651850316 13469622966 1817772651
   FAT32 LBA             11659936507 13664867953 2004931447
   FAT32 LBA             11666009078 14670808329 3004799252

[ Continue ]
(many more lines...)

any idea what else to do?

thanks
Daniel

S.Haran · 07-12-2014, 12:20 PM

Check the SMART info on the drives perhaps the drives are failing. The Linux command to use is smartctl.

Seeing i/o errors with your hexeditor is not a good sign. Try the Linux hexedit command. Also try hexedit on another spare device or binary file so you can see what to expect.

The purpose of this is to verify the decryption has worked. If you can not find any human readable text in the hexeditor then it is likely the decryption has failed.

DeepB · 07-13-2014, 05:17 AM

I can open /dev/md9 with a hex editor, but not /dev/mapper/data.

Smart says for both remaining disks:

Code:

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

Is that enough or should I run a special test?

thanks

S.Haran · 07-13-2014, 08:43 AM

In SMART see the values for Offline_Uncorrectable, Reallocated_Sector_Ct and Current_Pending_Sector. It's best if they are zero. You could also run a short and long test with smartctl.

But the first problem you need to solve is opening /dev/mapper/data.

Try creating another encrypted device as a test and see if you can open it with hexedit.

DeepB · 07-13-2014, 12:55 PM

Yes, I created another encrypted file, opened it, and i can read /dev/mapper/test without a problem in a hexeditor. (cryptsetup luksOpen /dev/loop0 test, then hte /dev/mapper/test)

S.Haran · 07-14-2014, 12:40 PM

You might look in /var/log/messages to see if there are any clues related to the problem you are having with /dev/mapper/data

DeepB · 07-14-2014, 01:28 PM

I have a lot of those:

Code:

Jul 13 11:58:36 server kernel: [233598.852774] sd 0:0:0:0: [sdd] CDB:
Jul 13 11:58:36 server kernel: [233598.852775] Read(16): 88 00 00 00 00 00 af e2 d4 d0 00 00 00 08 00 00
Jul 13 12:03:59 server kernel: [233921.370020] sd 0:0:0:0: [sdd] Unhandled sense code
Jul 13 12:03:59 server kernel: [233921.370037] sd 0:0:0:0: [sdd]
Jul 13 12:03:59 server kernel: [233921.370041] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 13 12:03:59 server kernel: [233921.370045] sd 0:0:0:0: [sdd]
Jul 13 12:03:59 server kernel: [233921.370048] Sense Key : Medium Error [current]
Jul 13 12:03:59 server kernel: [233921.370054] Info fld=0xafe32aa8
Jul 13 12:03:59 server kernel: [233921.370057] sd 0:0:0:0: [sdd]
Jul 13 12:03:59 server kernel: [233921.370060] Add. Sense: Unrecovered read error

S.Haran · 07-14-2014, 04:14 PM

That could be the problem. It looks like sdd is failing with read errors, likely due to bad sectors.

You could try to image it and if you can make a good image. Use that to reassemble the RAID5.

But if this is high value data you may want to seek pro help.

DeepB · 07-19-2014, 03:17 AM

The data is not valuable enough. I would however really like to try everything to get it back that is easily doable.

Any more ideas?

thank you

DeepB · 08-16-2014, 04:10 AM

any more ideas?

thanks