Need help to recover luks partition

Unfortunately, I have an infamous motherboard which has HPA and was responsible for this situation.

I didn't knew about the HPA curse until now, because of this, and learnt the hard way.

Anyone interested on HPA curse can read more here in detail:
https://forums.lime-technology.com/t...fferent-sizes/


So, to make a long story short, I have a 3TB disk which was partitioned like this:
parted /dev/sdx
mklabel gpt
unit s
mkpart primary ext3 2048s -1s
Warning: You requested a partition from 2048s to 5860533167s.
The closest location we can manage is 2048s to 5860533134s.
Is this still acceptable to you?
Yes/No? y

then:
cryptsetup luksFormat --cipher aes-cbc-essiv:sha256 --verify-passphrase --key-size 256 /dev/sdx1
cryptsetup luksOpen /dev/sdx1 sdx1_crypt
mkfs.ext3 /dev/mapper/sdx1_crypt

Afterwards, I started using this disk for storing ebooks, movies, pictures, etc, although most of the time it was just in a drawer.

Everything went fine during many months, until one day it stopped working because of the HPA curse I wrote about above, when I connected it to my main computer and found there was nothing in it; no LUKS partition, nothing.

I remembered one time I had a similar situation and it was solved by creating an exactly equal new partition and so I did it, but it didnt work this time:
parted /dev/sdx
mklabel gpt
unit s
mkpart primary ext3 2048s -1s
Warning: You requested a partition from 2048s to 5860533167s.
The closest location we can manage is 2048s to 5860531021.
Is this still acceptable to you?
Yes/No? y

But this time I noticed that the last sector had changed and the total number was less 2113 sectors!

At that time, I didn't knew why this 2113 sectors were missing; now I know is because of the HPA curse.

So, I stopped and made a clone of this disk using DD to another disk of same make, model and capacity that fortunately I had around.

Then I copied the partition table of a similar disk I had (also partitioned and formatted the same way and of same make, model and capacity)to this clone:
dd if=/dev/zero of=/dev/sdb bs=512 count=1
but I couldn't recover my luks partition

So, then I've ran testdisk twice:
one for EFI or GPT partition structure and a second time for intel partition structure just to see the differences.

Here's the output of the GPT partition structure (if the second is also needed, please say so and I'll post it):

TestDisk 6.13, Data Recovery Utility, November 2011
Christophe GRENIER <grenier@cgsecurity.org>
http://www.cgsecurity.org

Disk /dev/sdb - 3000 GB / 2794 GiB - CHS 364801 255 63

The harddisk (3000 GB / 2794 GiB) seems too small! (< 14785030 TB / 13446906 TiB)
Check the harddisk size: HD jumpers settings, BIOS detection...

The following partition can't be recovered:
Partition Start End Size in sectors
> MS Data 642191996 28877012508677537 28877011866485541


[ Continue ]
JFS 1381173050, 14785030 TB / 13446906 TiB

TestDisk 6.13, Data Recovery Utility, November 2011
Christophe GRENIER <grenier@cgsecurity.org>
http://www.cgsecurity.org

Disk /dev/sdb - 3000 GB / 2794 GiB - CHS 364801 255 63
Partition Start End Size in sectors
>P MS Data 3298905435 3302322120 3416686 [~UM-X D ^X^\ M-3Cm]

Structure: Ok. Use Up/Down Arrow keys to select partition.
Use Left/Right Arrow keys to CHANGE partition characteristics:
P=Primary D=Deleted
Keys A: add partition, L: load backup, T: change type,
Enter: to continue
cramfs, 1749 MB / 1668 MiB

Testdisk.log

Tue Sep 5 14:26:21 2017
Command line: TestDisk

TestDisk 6.13, Data Recovery Utility, November 2011
Christophe GRENIER <grenier@cgsecurity.org>
http://www.cgsecurity.org
OS: Linux, kernel 3.2.0-4-amd64 (#1 SMP Debian 3.2.81-2) x86_64
Compiler: GCC 4.6
Compilation date: 2012-01-17T14:04:23
ext2fs lib: 1.42.5, ntfs lib: 10:0:0, reiserfs lib: none, ewf lib: none
/dev/sdb: LBA, HPA, LBA48, DCO support
/dev/sdb: size       5860533168 sectors
/dev/sdb: user_max   5860533168 sectors
/dev/sdb: native_max 5860533168 sectors
Hard disk list
Disk /dev/sdb - 3000 GB / 2794 GiB - CHS 364801 255 63, sector size=512 - ST3000DM001-1CH166, S/N:Z1F0TKXK, FW:CC43

Partition table type (auto): EFI GPT
Disk /dev/sdb - 3000 GB / 2794 GiB - ST3000DM001-1CH166
Partition table type: EFI GPT

Analyse Disk /dev/sdb - 3000 GB / 2794 GiB - CHS 364801 255 63
hdr_size=92
hdr_lba_self=1
hdr_lba_alt=5860533167 (expected 5860533167)
hdr_lba_start=34
hdr_lba_end=5860533134
hdr_lba_table=2
hdr_entries=128
hdr_entsz=128
check_part_gpt failed for partition
 1 P MS Data                     2048 5860533134 5860531087 [3TBbk]
Current partition structure:
No FAT, NTFS, ext2, JFS, Reiser, cramfs or XFS marker
 1 P MS Data                     2048 5860533134 5860531087 [3TBbk]
 1 P MS Data                     2048 5860533134 5860531087 [3TBbk]

search_part()
Disk /dev/sdb - 3000 GB / 2794 GiB - CHS 364801 255 63

recover_JFS: s_blocksize=424374330
recover_JFS: s_size 8234812537959046209
recover_JFS: s_fsckpxd.len:13554014
recover_JFS: s_logpxd.len:4479763
recover_JFS: part_size 28877011866485541
     MS Data                642191996 28877012508677537 28877011866485541
     JFS 1381173050, 14785030 TB / 13446906 TiB
This partition ends after the disk limits. (start=642191996, size=28877011866485541, end=28877012508677537, disk end=5860533168)

SYSV4 Marker at 41949/41/20

recover_sysv4
     Unknown                673913287 2171750072774 2171076159488 [B'’ƒfª]
     SysV4, 1111 TB / 1010 TiB
Partition not added.
check_FAT: Bad jump in FAT partition

cramfs Marker at 205347/93/22

recover_cramfs
     MS Data               3298905435 3302322120    3416685 [•ôØßSDö{ðûN³Cm]
     cramfs, 1749 MB / 1668 MiB
check_FAT: Bad jump in FAT partition

LVM magic value at 239851/51/9
check_FAT: Bad number of sectors per cluster

LVM magic value at 290039/129/30
Disk /dev/sdb - 3000 GB / 2794 GiB - CHS 364801 255 63
Check the harddisk size: HD jumpers settings, BIOS detection...
The harddisk (3000 GB / 2794 GiB) seems too small! (< 14785030 TB / 13446906 TiB)
The following partition can't be recovered:
     MS Data                642191996 28877012508677537 28877011866485541
     JFS 1381173050, 14785030 TB / 13446906 TiB

Results
   P MS Data               3298905435 3302322120    3416686 [•ôØßSDö{ðûN³Cm]
     cramfs, 1749 MB / 1668 MiB
SIGINT detected! TestDisk has been killed.

Maybe this is also useful:

00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001c0  01 00 ee fe ff ff 01 00  00 00 ff ff ff ff 00 00  |................|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000200  45 46 49 20 50 41 52 54  00 00 01 00 5c 00 00 00  |EFI PART....\...|
00000210  6a 10 1f 57 00 00 00 00  01 00 00 00 00 00 00 00  |j..W............|
00000220  4d 93 50 5d 01 00 00 00  22 00 00 00 00 00 00 00  |M.P]....".......|
00000230  2c 93 50 5d 01 00 00 00  c5 c7 67 95 55 51 cb 49  |,.P]......g.UQ.I|
00000240  ae 6b 5e 65 13 28 19 69  02 00 00 00 00 00 00 00  |.k^e.(.i........|
00000250  80 00 00 00 80 00 00 00  86 d2 54 ab 00 00 00 00  |..........T.....|
00000260  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00002800

Is it still possible to recover this luks partition?

Argggh !.
I'm thinking you have lost the lot - have a read of this post. Hopefully rknichols will drop by and offer an opinion.

Hello

The sad news is that LUKS stores the encrypted master key in a header at the beginning of the container (partition/disk/LVM volume/...). That header is 1 to 2 MB in size (depends on the cipher's key size). Once even one sector of that key material is overwritten, the master key is forever unrecoverable.

Overwriting the start of the disk with a copy of the BIOS would certainly do that to a partition in that region. But even had the BIOS not done that, running "mkpart primary ext3 2048s -1s" would also have written an ext3 super block and group descriptor there, overwriting the LUKS header in that partition. You would have to tell parted to create an empty partition, not a filesystem, in order to preserve the LUKS header.

You can tell fairly quickly whether there is any chance of recovery. Run "hexedit -s /dev/sdx" and search for the hex string "4C 55 4B 53 BA BE" at the start of a sector. (That's the ASCII string "LUKS" followed by the hex bytes 0xBA and 0xBE.) If you don't find that within the first few megabytes of the disk, the LUKS header is gone. Without a backup of that header there is no chance of recovering any data. (Running luksFormat with the same passphrase will not help. The actual master key is a random number unrelated to the passphrase.)

Welcome to the club. Almost everyone who has used encryption has at some point lost access to some of their data. (Yes, me too. Why did I think the index to a collection of naughty magazines was so private when the mags themselves were right over there in a box?) It's like the lock on a vault, deliberately designed to be as unforgiving as possible if everything is not exactly right.

Should "HDA" be "HPA"? Anyway, reading around about it, I think the end of the disc gets overwritten, not the start. So, if the important stuff is stored near the start, then there may be hope?

Originally Posted by syg00 Argggh !. I'm thinking you have lost the lot - have a read of this post

Ok, thanks. I've read it and understood it

Originally Posted by rknichols The sad news is that LUKS stores the encrypted master key in a header at the beginning of the container (partition/disk/LVM volume/...). That header is 1 to 2 MB in size (depends on the cipher's key size). Once even one sector of that key material is overwritten, the master key is forever unrecoverable. Overwriting the start of the disk with a copy of the BIOS would certainly do that to a partition in that region. But even had the BIOS not done that, running "mkpart primary ext3 2048s -1s" would also have written an ext3 super block and group descriptor there, overwriting the LUKS header in that partition. You would have to tell parted to create an empty partition, not a filesystem, in order to preserve the LUKS header. You can tell fairly quickly whether there is any chance of recovery. Run "hexedit -s /dev/sdx" and search for the hex string "4C 55 4B 53 BA BE" at the start of a sector. (That's the ASCII string "LUKS" followed by the hex bytes 0xBA and 0xBE.) If you don't find that within the first few megabytes of the disk, the LUKS header is gone. Without a backup of that header there is no chance of recovering any data. (Running luksFormat with the same passphrase will not help. The actual master key is a random number unrelated to the passphrase.)

Thank you for this very clear explanation.
I've searched the hex string but unfortunately it just wasn't there

Originally Posted by IsaacKuo Should "HDA" be "HPA"?

Yes, HPA. Sorry, my bad.
Thanks for pointing that out. I've edited the OP.

Originally Posted by IsaacKuo reading around about it, I think the end of the disc gets overwritten, not the start. So, if the important stuff is stored near the start, then there may be hope?

Unfortunately no because the start also got a litle overwritten, as rknichols explained here:

Originally Posted by rknichols Overwriting the start of the disk with a copy of the BIOS would certainly do that to a partition in that region. But even had the BIOS not done that, running "mkpart primary ext3 2048s -1s" would also have written an ext3 super block and group descriptor there, overwriting the LUKS header in that partition. You would have to tell parted to create an empty partition, not a filesystem, in order to preserve the LUKS header.

I guess I'm just going to accept defeat and live with the last backup I have.
I've lost about 10% of data from the time it was made.
Could be a lot worst

Thank you all for helping me!

Now I know the solution for this situation was quite simple.
In fact, all I had to do was remove the HPA and the LUKS partition and data would come back after restarting the computer:

1st step - check if HPA is enabled:
hdparm -N /dev/sdx

/dev/sdx:
max sectors = 78125000/78165360, HPA is enabled

2nd step - remove the HPA:
hdparm -N p"max number of sectors" /dev/sdx

example:
hdparm -N p78165360 /dev/sdx
78165360 = denominator from fraction above

done
restart

credits:
https://superuser.com/questions/6426...google_rich_qa

see also DCO

Just a footnote, when a bios places a HPA on a disk it does so in a nonpersistent way. With each power cycle the bios has to put it back again. So, you can try to do a drive reset with hdparm, which should get rid of the HPA. Or, you can hot cycle the power to the drive by disconnecting the power for several seconds and reconnecting it with the machine powered up.

This works because it power cycles the drive, getting rid of the HPA, but the BIOS has no opportunity to put it back. But I also fear that the luks header is gone. Hopefully the next person will read the entire thread before messing around with the drive.