Hi

Forgive my English, it's not my mother tongue I am Italian.
It 's my first post in this forum so be nice :-)

I have the following problem:
On an Ubuntu system an active encrypted partition was active with cryptsetup for 7-year on RAID6 /dev/md0.
Now after having some problems with the RAID, that I think I have solved I can not access the encrypted partition.

# cryptsetup -v isLuks /dev/md0
Device /dev/md0 is not a valid LUKS device.
Command failed with code 22: Device /dev/md0 is not a valid LUKS device.

The RAID6 is working:

# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Sat Dec 9 18:07:44 2017
Raid Level : raid6
Array Size : 11717001216 (11174.20 GiB 11998.21 GB)
Used Dev Size : 1952833536 (1862.37 GiB 1999.70 GB)
Raid Devices : 8
Total Devices : 8
Persistence : Superblock is persistent

Intent Bitmap : Internal

Update Time : Sat Dec 9 18:07:44 2017
State : clean
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 512K

Name : nebula:0 (local to host nebula)
UUID : efc2e139:1eafbc2f:f8015242:1b0d6687
Events : 0

Number Major Minor RaidDevice State
0 8 49 0 active sync /dev/sdd1
1 8 65 1 active sync /dev/sde1
2 8 81 2 active sync /dev/sdf1
3 8 97 3 active sync /dev/sdg1
4 8 113 4 active sync /dev/sdh1
5 8 129 5 active sync /dev/sdi1
6 8 145 6 active sync /dev/sdj1
7 8 161 7 active sync /dev/sdk1


I have the following data, saved when I created the partition, but I do not have the backup binary file.
Is it possible using these data, and obviously the password for the key slot 0 restore the header?
I'm quite sure that there is no solution, but before reformat the RAID I wanted the opinion of someone more experienced.

# cryptsetup luksDump /dev/md0
LUKS header information for /dev/md0

Version: 1
Cipher name: aes
Cipher mode: cbc-essiv:sha256
Hash spec: sha256
Payload offset: 6144
MK bits: 256
MK digest: 1b 0d c7 43 38 59 03 da b6 02 38 45 1c 7e 24 15 41 77 a5 c3
MK salt: d7 33 ae 9f 05 4d d3 a4 d4 94 07 02 64 1c 5c 18
bf 23 e0 84 20 98 57 5e 57 57 d7 f8 2e dd 85 0a
MK iterations: 22875
UUID: 323b1ec2-ad9e-4501-a154-f8173d878b50

Key Slot 0: ENABLED
Iterations: 91580
Salt: 2a ee 98 33 f7 94 21 15 2a 50 17 ae 73 66 27 73
45 68 04 6a 13 52 5d fa 24 db 6e a5 ea 0f 55 ef
Key material offset: 8
AF stripes: 4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED

Thanks in advance.

Best regards

Adriano

It depends on how much was overwritten. You've got enough to recover the header sector itself, but the actual key material for Key Slot 0 begins at sector 8 of that partition and occupies 250 sectors. If any of those sectors were overwritten, no recovery is possible.

You can look with a hex editor and see if sectors 8 through 257 contain random-appearing data. If they do, there is a good chance for recovery.

The passphrase is not needed to restore the header. You just need that to unlock the container once the header has been restored.

Many thanks for your help.

Unfortunatly in a matter of 2 seconds 6 of the 8 drives failed with not physical faults on the HD.
I think the problem was caused by a bus glitch and the linux RAID start to recovery the fault drive.
In two second stop because there is too mach drives fault.

In order to reassemble the RAID, after a lot of tentative i had used this command, than I am sure was used in the original configuration:

#mdadm --create /dev/md0 --metadata=1.2 --auto=md --assume-clean --bitmap=internal --bitmap-chunk=131072 --chunk=512 --level=6 --raid-devices=8 /dev/sd[defghijk]1
mdadm: array /dev/md0 started

But when I try to unlock the partition in order to see the LVM volume I get the error.

I'm a bit rusty, but with a little help I think I can edit the sectors, if I know what to write to.
What do you think is the best procedure?

I would start by making a copy of the sectors that we want to modify and then I would try to rewrite them.

Many thanks for any help.

Best regards

Adriano

If you do find that the key material appears to be intact (i.e., suitably random), here's a uuencoded copy of what should be in the first 1024 bytes of /dev/md0.

Hi rknichols

After analyzing this data recovered from the /dev/sdd1 I am quite sure it's the complete LUKS header
(I select only 592 byte from 4096 the bytes beyond which the random data start, many of the bytes between 592 and 4096 are 00)


00000000 4C 55 4B 53 BA BE 00 01 61 65 73 00 00 00 00 00 LUKS....aes.....
00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000020 00 00 00 00 00 00 00 00 63 62 63 2D 65 73 73 69 ........cbc-essi
00000030 76 3A 73 68 61 32 35 36 00 00 00 00 00 00 00 00 v:sha256........
00000040 00 00 00 00 00 00 00 00 73 68 61 32 35 36 00 00 ........sha256..
00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000060 00 00 00 00 00 00 00 00 00 00 18 00 00 00 00 20 ...............
00000070 1B 0D C7 43 08 59 03 DA B6 02 38 55 1C 7E 24 15 ...C.Y....8U.~$.
00000080 41 77 A5 C3 D7 33 AE 9F 05 4D F3 A4 D4 94 07 02 Aw...3...M......
00000090 64 1C 5C 18 BF 23 E0 84 20 98 57 5E 77 57 D7 F8 d.\..#.. .W^wW..
000000A0 6E DD 85 0A 00 00 59 5B 33 32 33 62 31 65 63 32 n.....Y[323b1ec2
000000B0 2D 61 64 39 65 2D 34 35 30 31 2D 61 31 35 34 2D -ad9e-4501-a154-
000000C0 66 38 31 37 33 64 38 37 38 62 35 30 00 00 00 00 f8173d878b50....
000000D0 00 AC 71 F3 00 01 65 BC 2A EE 98 33 F7 94 21 15 ..q...e.*..3..!.
000000E0 DA 50 17 AE 73 66 27 73 45 68 04 6A 15 52 5D FA .P..sf'sEh.j.R].
000000F0 C4 DB 6E A5 EA 0F 55 EF 00 00 00 08 00 00 0F A0 ..n...U.........
00000100 00 00 DE AD 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000110 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000120 00 00 00 00 00 00 00 00 00 00 01 08 00 00 0F A0 ................
00000130 00 00 DE AD 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000140 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000150 00 00 00 00 00 00 00 00 00 00 02 08 00 00 0F A0 ................
00000160 00 00 DE AD 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000170 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000180 00 00 00 00 00 00 00 00 00 00 03 08 00 00 0F A0 ................
00000190 00 00 DE AD 00 00 00 00 00 00 00 00 00 00 00 00 ................
000001A0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
000001B0 00 00 00 00 00 00 00 00 00 00 04 08 00 00 0F A0 ................
000001C0 00 00 DE AD 00 00 00 00 00 00 00 00 00 00 00 00 ................
000001D0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
000001E0 00 00 00 00 00 00 00 00 00 00 05 08 00 00 0F A0 ................
000001F0 00 00 DE AD 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000210 00 00 00 00 00 00 00 00 00 00 06 08 00 00 0F A0 ................
00000220 00 00 DE AD 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000230 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000240 00 00 00 00 00 00 00 00 00 00 07 08 00 00 0F A0 ................

thath mach with my original

# cryptsetup luksDump /dev/md0
LUKS header information for /dev/md0

Version: 1
Cipher name: aes
Cipher mode: cbc-essiv:sha256
Hash spec: sha256
Payload offset: 6144
MK bits: 256
MK digest: 1b 0d c7 43 08 59 03 da b6 02 38 55 1c 7e 24 15 41 77 a5 c3
MK salt: d7 33 ae 9f 05 4d f3 a4 d4 94 07 02 64 1c 5c 18
bf 23 e0 84 20 98 57 5e 77 57 d7 f8 6e dd 85 0a
MK iterations: 22875
UUID: 323b1ec2-ad9e-4501-a154-f8173d878b50

Key Slot 0: ENABLED
Iterations: 91580
Salt: 2a ee 98 33 f7 94 21 15 da 50 17 ae 73 66 27 73
45 68 04 6a 15 52 5d fa c4 db 6e a5 ea 0f 55 ef
Key material offset: 8
AF stripes: 4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED

How can I try to restore the header now hoping then to be able to see after the LVM volumes?

Best regards

Adriano

Aside from a few typos I made copying data from your original post (I've fixed those), that is the same as what I came up with and should be a good LUKS header. What do you see if you look at the first sectors of /dev/md0? Perhaps the array is not really assembled the way it was before.

Hi

This is /dev/mb0 now.


00000000 A5 64 E3 BE 1F 42 A0 57 7C 47 6C 94 E5 03 A1 9E .d...B.W|Gl.....
00000010 AD 42 D2 1B 0D DD 10 B0 20 95 19 46 B0 F4 E0 A2 .B...... ..F....
00000020 B6 D0 6A 61 BC BC 49 F8 62 33 3E 65 AF 06 5F 3B ..ja..I.b3>e.._;
00000030 37 0B 97 62 67 D9 B2 68 2B 40 2F CC 1C 69 55 CD 7..bg..h+@/..iU.
00000040 6E A4 43 A3 E0 B9 CA C0 49 7A D0 99 5C 1A EA A4 n.C.....Iz..\...
00000050 21 14 82 F0 73 5F BE 66 78 58 66 8C 11 00 96 3C !...s_.fxXf....<
00000060 0F 5C C1 E2 14 3B F4 AE FC 9B 96 F9 13 1B 54 CD .\...;........T.
00000070 86 B8 E7 D4 82 94 03 B8 D8 D9 95 B4 91 FC B6 FF ................
00000080 EF 23 AD 43 43 EA 57 E8 3E CE C7 3C C3 95 B3 38 .#.CC.W.>..<...8
00000090 59 00 7D 75 E7 28 E6 2E A2 A7 E0 29 06 05 24 C5 Y.}u.(.....)..$.
000000A0 9C D3 14 EB 91 95 C2 B9 80 C1 3C 52 FE F3 B4 4D ..........<R...M
000000B0 BB AC 57 2D 86 3C 97 3B 12 3D 1C 50 A0 E9 91 97 ..W-.<.;.=.P....
000000C0 F5 95 41 D7 60 51 87 15 48 7F FD F2 6D 0B F6 36 ..A.`Q..H...m..6
000000D0 FB 37 E7 F1 92 4D D5 8F 86 57 5E D1 7E 0C 44 1D .7...M...W^.~.D.
000000E0 E5 D5 9A EE AB 12 DA 2D 41 D7 C1 F4 F4 67 61 C8 .......-A....ga.
000000F0 25 71 78 1A 81 B2 78 24 EF 0E 5B 9C 10 9D 7C 84 %qx...x$..[...|.
00000100 DC 06 E8 63 A0 CD 68 0B 54 3D E2 C5 14 6F 31 B4 ...c..h.T=...o1.
00000110 4D CB 26 95 B7 09 66 74 1C 9D 91 E4 CF 38 CB 12 M.&...ft.....8..
00000120 52 91 37 73 AD 06 D6 5D E8 27 17 C6 45 B8 48 45 R.7s...].'..E.HE
00000130 F6 8E ED 48 2D 74 E9 0A 86 A1 09 BB F9 E8 DD DE ...H-t..........
00000140 1B E4 0F 2A 4E FC 28 E4 A9 53 74 12 6D FC BA D1 ...*N.(..St.m...
00000150 FA 6F AA CB 98 1B 62 CB 5B 78 07 C3 1A 87 E6 3F .o....b.[x.....?
00000160 C8 BF A4 36 22 95 43 F8 D1 FD 86 DB 1B 5A B1 B5 ...6".C......Z..
00000170 92 8B 80 99 13 EE C9 BE 4F B4 D8 E1 2D 71 B2 75 ........O...-q.u
00000180 19 11 A1 D4 53 5D 5D 20 F5 78 3F 59 15 06 01 A7 ....S]] .x?Y....
00000190 95 F5 BE DE 62 E0 B5 CB 9F 77 3D 96 A4 19 45 3C ....b....w=...E<
000001A0 C1 2A 13 2A 82 D3 1B BC 5B 9D 32 6E 84 24 5D 27 .*.*....[.2n.$]'
000001B0 06 9A 1C 18 66 B5 A8 4C E5 8A 59 02 64 96 40 B4 ....f..L..Y.d.@.
000001C0 5C F7 1C F5 66 B0 58 B5 99 1C 87 E4 C7 6B A1 90 \...f.X......k..
000001D0 61 8A 7E 1E E9 F1 6F 90 F1 B8 12 DF 54 8D A8 11 a.~...o.....T...
000001E0 B8 7C 05 09 96 33 36 60 44 14 F4 0F 64 15 C5 49 .|...36`D...d..I
000001F0 E5 0D 12 8A 65 11 30 5D 68 45 44 D7 95 82 B1 0F ....e.0]hED.....
00000200 07 81 EF CD D5 06 25 63 3E DC 05 9C E7 59 BC CB ......%c>....Y..
00000210 AB E8 DA 54 9C 47 E4 E1 94 60 1C 8D 9A D4 4A 78 ...T.G...`....Jx
00000220 1C D8 9D 52 80 83 CC 23 DF AE 34 41 D4 DE 0A 76 ...R...#..4A...v
00000230 6A 6C 8E 40 3E 1A 5F 6D 7F 00 6D ED DF 9E 52 69 jl.@>._m..m...Ri
00000240 DC FF 5B 37 16 B7 96 9C 00 F5 1D 80 06 27 CB A6 ..[7.........'..
00000250 C9 DC 36 D4 45 4E 45 34 9C AA F5 DB DC B1 77 9E ..6.ENE4......w.
00000260 E0 39 B4 06 AB D8 95 83 15 F8 D7 7E 57 F5 20 16 .9.........~W. .
00000270 95 94 20 69 42 ED F7 13 2F 79 32 D3 11 89 D5 74 .. iB.../y2....t
00000280 1E 17 3A 03 74 16 1E BC 9E 60 FA 4A BC B4 8C 50 ..:.t....`.J...P
00000290 8D 2E 6F 17 BC D8 42 AF 55 65 53 71 45 E8 DE B7 ..o...B.UeSqE...
000002A0 F4 5C 67 E1 0F 39 2A C4 62 A0 31 EE 74 2E 41 25 .\g..9*.b.1.t.A%
000002B0 E2 C0 19 87 14 5A 43 1E 12 61 B6 C6 47 8A 9D 88 .....ZC..a..G...
000002C0 E1 8E E7 0C 33 99 B9 5D 5C 40 FE DF A9 E3 03 17 ....3..]\@......
000002D0 84 99 F3 5B 86 CE B2 52 69 B0 5C E9 1B 21 41 5C ...[...Ri.\..!A\
000002E0 04 A3 A7 22 B6 7B 97 A5 BD 14 26 C3 61 2C 04 86 ...".{....&.a,..
000002F0 B3 97 43 E3 38 5E 23 2B A1 B5 7C 0A 14 33 60 38 ..C.8^#+..|..3`8
00000300 BD B9 81 91 B9 B6 6F E9 85 70 78 46 99 31 A4 77 ......o..pxF.1.w
00000310 EF E5 EB 40 A4 9E 58 A3 50 8A 15 F5 0F 26 A6 B3 ...@..X.P....&..
00000320 C9 20 7C 79 8F 32 A0 01 BD CF 51 7F 72 B0 87 44 . |y.2....Q.r..D
00000330 97 15 47 F3 93 B1 6D 1B 67 65 4D F3 56 F0 6F A2 ..G...m.geM.V.o.
00000340 22 5B AA 5D 1C D7 6A 82 DB BB 78 31 53 BC C5 51 "[.]..j...x1S..Q
00000350 09 C6 E9 8B F7 D2 51 A1 C5 58 63 05 62 B9 E7 0E ......Q..Xc.b...
00000360 D6 EA B4 BA 15 19 83 84 4A 3F A5 12 4A 8D 86 92 ........J?..J...
00000370 AA BD CA 38 54 F3 74 A3 86 3C 5B 6D 53 20 15 9F ...8T.t..<[mS ..
00000380 71 15 2E 0D E6 13 9A DF 2E FA 48 5A E9 69 8C 76 q.........HZ.i.v
00000390 E2 6C BD 2F ED 5E 65 D9 42 A4 54 35 20 BD 6C 13 .l./.^e.B.T5 .l.
000003A0 B5 36 A5 93 42 C3 57 64 D5 E7 97 31 FE 38 32 F2 .6..B.Wd...1.82.
000003B0 CE B3 AF 78 A6 8E 5C 4D F0 75 0F C5 AC 93 B0 BD ...x..\M.u......
000003C0 B9 AF 0B 5B 74 27 FA A5 F6 04 30 C8 4B AB 9E 2D ...[t'....0.K..-
000003D0 5F 27 E8 C6 09 DE C9 56 6D A5 97 A2 9B 83 21 8D _'.....Vm.....!.
000003E0 E5 6B 0F 5D 29 D9 19 FF 7D AB 56 4B 47 66 76 DC .k.])...}.VKGfv.
000003F0 56 67 3E 82 0D 5D 74 12 2A 0F 7A 41 50 D2 63 F8 Vg>..]t.*.zAP.c.
00000400 94 24 90 27 21 1D C4 A1 F5 45 51 10 F7 3B 02 E6 .$.'!....EQ..;..
00000410 36 83 13 29 21 58 D7 39 69 27 24 4A E8 18 5B AC 6..)!X.9i'$J..[.
00000420 CE 18 A2 76 36 1A 30 F6 B1 34 3B 9C 78 C3 50 3A ...v6.0..4;.x.P:
00000430 65 97 00 7D 34 80 00 8A 38 24 B7 EB 9F 47 6B 1B e..}4...8$...Gk.
00000440 80 FE A0 53 A2 A3 26 75 6D B6 27 78 E2 1C E8 EB ...S..&um.'x....
00000450 69 6E F6 FE 98 FC 3C 05 FF EA 43 B7 5C 69 B9 EC in....<...C.\i..
00000460 53 51 C7 C4 9C 4F 31 2D 60 F9 D2 A5 7A 94 14 E4 SQ...O1-`...z...
00000470 EA D3 91 AF F7 D7 D3 2E 38 B0 7E 7C D0 E9 51 1C ........8.~|..Q.
00000480 B1 68 A9 22 A7 C6 38 DE FC 8F 9C 07 1B 62 04 9D .h."..8......b..
00000490 BD 9F 5C AC EB BD 5F 1C F5 08 37 D4 A3 D6 76 DD ..\..._...7...v.
000004A0 23 2D 5A 7B 5D B7 E7 DE F0 B8 DF 1A 15 C1 4C CE #-Z{].........L.
000004B0 A0 7A E8 6E 45 BD DD 71 73 12 07 53 2A 94 F7 F0 .z.nE..qs..S*...
000004C0 E4 4F DB 89 06 20 5A 31 C5 DA D7 19 7D 9B 53 17 .O... Z1....}.S.
000004D0 3F 73 A4 4B 23 22 73 34 47 99 F2 6D 09 8C 88 1D ?s.K#"s4G..m....
000004E0 42 E8 FD D6 6D 93 C9 C0 E0 CA 3B BF CD B5 CC E3 B...m.....;.....
000004F0 5F E4 61 72 2A D5 99 D6 24 72 F5 C4 56 54 6E CF _.ar*...$r..VTn.
00000500 60 E5 44 21 56 12 46 71 50 1F 1C E9 90 8B A6 63 `.D!V.FqP......c
00000510 5F B9 8C 1A 8F 07 64 12 EE C1 76 48 B2 CB 38 E6 _.....d...vH..8.
00000520 00 4B F2 64 63 53 6C B7 D6 9C 78 29 C9 E5 58 0A .K.dcSl...x)..X.
00000530 25 FC 32 67 E3 C0 46 E2 19 49 5E 53 E4 71 6F D3 %.2g..F..I^S.qo.
00000540 B3 3D D4 19 D9 A5 27 FF 21 5B 4F 47 BD E6 9F DD .=....'.![OG....
00000550 81 55 02 17 C8 38 BC DC 52 8F 04 9D 46 97 F1 14 .U...8..R...F...
00000560 4C E0 49 02 B7 6D 72 76 C8 17 5B F7 FB F5 5D E9 L.I..mrv..[...].
00000570 98 94 7E 77 60 10 E3 E5 96 BD 46 DD 98 5B 90 01 ..~w`.....F..[..
00000580 BA C1 5E D2 4C 9A A6 8A 4F F9 67 40 98 1E 01 45 ..^.L...O.g@...E
00000590 54 FD 58 DF 03 EA 17 B4 9B C2 B4 EF 48 CC 37 AE T.X.........H.7.
000005A0 61 72 BC A2 44 DF 19 F0 8F 7A F5 C4 DD 6D 73 F9 ar..D....z...ms.
000005B0 BE 07 6C D1 20 EA D4 FA E7 7B 9A BC D3 E5 30 CE ..l. ....{....0.
000005C0 18 2D B3 66 B2 05 91 CE 4E 9B 8C 6B 90 40 CA 4E .-.f....N..k.@.N
000005D0 30 04 28 D7 EC 08 D0 D6 52 5E 3C 09 43 A3 C0 68 0.(.....R^<.C..h
000005E0 89 D8 C7 5F 8C A4 58 55 34 10 57 C5 2C 27 D3 21 ..._..XU4.W.,'.!
000005F0 D5 D9 50 12 70 3E F5 2A 2C 38 DD E8 E6 AD 50 51 ..P.p>.*,8....PQ
00000600 ED F1 5D 45 DB 49 C3 BE 3E C5 BB D0 80 FD 88 7C ..]E.I..>......|
00000610 13 7A F4 EC 90 BF 4F B8 9D F5 B4 3D B1 C7 34 AA .z....O....=..4.
00000620 D2 42 DB DA FC 72 9C 79 48 AD 96 9A B8 FF 6C 86 .B...r.yH.....l.
00000630 98 AC D8 58 4C 6E E8 EE 6B 61 7F 05 27 8E C4 56 ...XLn..ka..'..V
00000640 8D 06 52 31 5D 72 E0 B6 56 9B FD 7D 1E E1 15 FD ..R1]r..V..}....
00000650 5D 4A 70 99 14 35 28 1A AC 67 9C 15 3D C9 84 33 ]Jp..5(..g..=..3
00000660 B9 74 2E 9E A0 DA 3B 59 80 D5 95 1C 1A FB 2B 0C .t....;Y......+.
00000670 5A 23 93 0A A7 72 10 BC 98 EA 4B 49 3E A5 78 7C Z#...r....KI>.x|
00000680 28 3B D3 C3 78 4D 9D F5 85 52 5C AB F6 50 93 A8 (;..xM...R\..P..
00000690 5C 0D F1 E8 5B 27 54 2C 48 6A 3B E1 AD 69 27 D3 \...['T,Hj;..i'.
000006A0 53 26 E4 B9 28 2D EA 49 FC E7 01 71 47 6A 02 5B S&..(-.I...qGj.[
000006B0 D1 04 4B D0 D6 38 B2 20 DF DC F6 FE D9 CB 3D 68 ..K..8. ......=h
000006C0 17 69 B7 02 B2 7D AE B4 36 3F 61 99 47 AB 1D 32 .i...}..6?a.G..2
000006D0 E8 F8 FB 17 D7 1A 92 65 1E 20 A9 EE AD 7E 99 EC .......e. ...~..
000006E0 5A 6D 73 ED 29 1E 71 1A BE 96 59 54 CE 34 89 68 Zms.).q...YT.4.h
000006F0 31 73 83 2C D3 7E 7B 87 13 25 AE 3C FE 8A 4F 76 1s.,.~{..%.<..Ov
00000700 08 63 BA E2 E0 20 28 15 00 AD BE EE C5 20 57 C8 .c... (...... W.
00000710 AB B6 DD 84 61 F6 B1 C0 0C 1E B8 A4 54 A1 AA 3B ....a.......T..;
00000720 41 E5 AD ED 20 F5 5C 04 44 51 D2 50 07 0E 80 FF A... .\.DQ.P....
00000730 38 00 82 14 CB 9C 47 B7 7C 65 54 7B 74 30 EB 65 8.....G.|eT{t0.e
00000740 08 BB CB 88 10 F0 75 90 6B E6 AB E6 75 37 6A C0 ......u.k...u7j.
00000750 F6 9D A4 D5 03 DB 60 40 2E 2C 36 0E 36 7F 0A 93 ......`@.,6.6...
00000760 1D 1F 71 A3 35 DF 7E BE E3 F8 07 42 01 D8 57 48 ..q.5.~....B..WH
00000770 6B 38 99 35 42 9E A8 B5 98 07 3F C5 BD 61 AE F1 k8.5B.....?..a..
00000780 B3 AF 83 81 AA 57 C9 08 46 FC 8D E1 01 BA 7C 59 .....W..F.....|Y
00000790 68 F5 BD 1C 69 7B 14 82 4D 47 49 A1 8F 50 96 0A h...i{..MGI..P..
000007A0 26 8A C9 66 1E E8 65 D4 60 7B AB EA DA CF 50 A9 &..f..e.`{....P.
000007B0 FD 40 C2 E0 5B B5 32 75 F6 C4 C7 28 DF 5A 5B BC .@..[.2u...(.Z[.
000007C0 CC 82 F2 CD 0B BD BA FF 29 D7 6F 16 35 F2 58 E1 ........).o.5.X.
000007D0 E1 A7 A4 47 AE D6 ED 03 10 93 8A 6E 3D F2 EB 4F ...G.......n=..O
000007E0 25 5D 86 ED EB D3 DD 9D F0 F3 CF 80 01 AD A4 8C %]..............
000007F0 79 48 87 92 5E 13 25 EB D5 7E 4E F7 17 9A 4B 45 yH..^.%..~N...KE
00000800 47 1B 93 D9 3C 63 A8 81 2F D2 E0 C9 47 80 4E D3 G...<c../...G.N.
00000810 DC 4E 1E DE 69 37 30 15 55 72 E2 76 8E 22 5E 51 .N..i70.Ur.v."^Q
00000820 4B 5B FE 8D DB 58 4D 90 F4 74 0F 01 8A 8D A5 0D K[...XM..t......
00000830 D2 C0 3C DE 12 6E 8D CE 76 C6 3B D9 F2 D2 88 48 ..<..n..v.;....H
00000840 43 CB D9 A0 9F 09 E8 DE 91 AD 72 E4 95 EF 1A 04 C.........r.....
00000850 B6 F3 1D 27 D0 14 42 47 64 D8 7F BD 47 CF B0 A3 ...'..BGd...G...
00000860 1F C5 45 2B 71 E8 86 C0 5C 22 67 FF 32 CD 57 62 ..E+q..."g.2.Wb
00000870 CF 04 06 CE 48 A1 75 6F 82 87 E8 25 79 DD 70 03 ....H.uo...%y.p.
00000880 84 90 79 4D AE 2C 17 09 1C 1E 44 46 01 1F 58 A1 ..yM.,....DF..X.
00000890 33 AB AA 82 75 27 F9 7E 3A 9A 4D D5 93 EB 0C A2 3...u'.~:.M.....
000008A0 E9 2B 4F 2A FF 9D 93 0F 9F B7 46 D3 A8 9D 5F C0 .+O*......F..._.
000008B0 67 FB 85 F8 E1 E4 88 84 58 FF CE 4D 34 E3 DE 96 g.......X..M4...
000008C0 72 72 81 A4 75 E2 15 17 AB B3 2C 0F 2F 8B 9B CA rr..u.....,./...
000008D0 14 66 B9 9F 55 C1 1B 46 4E 5A 71 2D 9C F3 FD 79 .f..U..FNZq-...y
000008E0 77 B6 CD 76 F8 AF EF 14 E7 93 3E D0 85 74 54 67 w..v......>..tTg
000008F0 C5 AC B5 1B D9 E3 A8 B8 3E 07 B1 89 2B B3 A9 70 ........>...+..p
00000900 FE 7E DB 26 20 A8 5A DC 96 C3 A1 73 02 B7 9B 79 .~.& .Z....s...y
00000910 1D 22 EB 5A EA 1C 5B A2 D4 44 26 8C B3 68 44 B4 .".Z..[..D&..hD.
00000920 5E 6C 46 C1 9D 21 60 3F 69 38 4B DE 2C 09 AD 2C ^lF..!`?i8K.,..,
00000930 52 83 D4 D5 7A 39 B1 A5 0D FE 89 38 5A EC CB 35 R...z9.....8Z..5
00000940 9E 30 80 B5 F8 D2 1F 8B F2 38 A4 8F BC 22 2C FF .0.......8...",.
00000950 C6 0E 03 37 81 A2 67 85 D1 C6 2E DF 06 CC A1 C5 ...7..g.........
00000960 2A B5 03 EC B8 AE B3 83 86 15 56 CC B7 05 39 8B *.........V...9.
00000970 2E 8E 60 42 AA 6B AD 44 1A F7 76 9D DF 20 90 8E ..`B.k.D..v.. ..
00000980 95 C3 F6 0C 95 BA 83 B7 5D 74 18 74 17 5D 3A BE ........]t.t.]:.
00000990 16 12 B5 95 67 5B BB 93 FF 27 1B 4E 90 26 28 D4 ....g[...'.N.&(.
000009A0 29 27 97 7F 65 19 FF C8 5D EA 4A 18 1B 84 55 D0 )'..e...].J...U.
000009B0 5A 62 AE E4 58 C4 31 E3 50 35 E3 18 EE DE 9D 00 Zb..X.1.P5......
000009C0 71 6D C1 DC 92 0E 83 EA A4 11 9C F9 A6 D2 27 B3 qm............'.
000009D0 71 43 6A B3 11 D1 16 43 9D 39 86 2C 86 E5 F2 08 qCj....C.9.,....
000009E0 54 40 BD 85 A4 D5 39 3D 42 42 70 A0 F9 F6 9E 3C T@....9=BBp....<
000009F0 75 86 9E 59 D0 0E D8 45 6E 0D D3 E6 78 30 6B 16 u..Y...En...x0k.
00000A00 CF 3E FB EA 62 6A A1 E9 B3 31 6D B6 80 43 FA B2 .>..bj...1m..C..
00000A10 46 43 70 F6 48 58 EF 63 81 3B E7 95 F5 5B 08 B9 FCp.HX.c.;...[..
00000A20 6D EB 99 1E 23 D1 04 8A C9 82 69 97 68 2B 17 92 m...#.....i.h+..
00000A30 3D C5 CD 6F A6 98 48 68 B8 94 9E 2F BE C7 A2 A7 =..o..Hh.../....
00000A40 74 9F 02 F0 EA 3A FB 47 C2 A8 14 6C A9 89 14 E7 t....:.G...l....
00000A50 FE A0 20 5A 42 CE 16 62 A4 BF 33 C9 61 48 8F 60 .. ZB..b..3.aH.`
00000A60 9F FD A9 3A F4 30 66 B2 30 79 77 F1 3D 8F AE C6 ...:.0f.0yw.=...
00000A70 C9 7E 03 23 5B 71 44 D6 87 06 16 C1 74 3A 8A B3 .~.#[qD.....t:..
00000A80 5E BB 58 41 65 C5 75 E8 FA C8 B0 01 13 EF B3 9D ^.XAe.u.........
00000A90 8E CD 30 F1 17 8E 43 14 C4 90 25 07 74 31 4D 1A ..0...C...%.t1M.
00000AA0 BC E9 F5 5E 2B 6E 7F 50 DB 60 D1 6D 39 6E 80 33 ...^+n.P.`.m9n.3
00000AB0 E4 A9 66 59 2F B9 54 07 04 B6 6C E7 28 EF FF 38 ..fY/.T...l.(..8
00000AC0 9E DE B9 9D 1B 28 27 F9 7A 62 72 7C 35 B0 E6 40 .....('.zbr|5..@
00000AD0 04 EE 73 F4 3D 37 C7 72 93 40 F4 75 43 53 8C 1B ..s.=7.r.@.uCS..
00000AE0 62 B7 CB A1 92 D2 E8 CA A5 D3 79 DA F9 62 28 A7 b.........y..b(.
00000AF0 4F D7 20 8A A6 C6 13 13 94 8B 99 B7 A2 07 0D 3F O. ............?
00000B00 2D 68 2F 34 3A 9D 6E B9 88 64 3C 39 35 FA 57 D7 -h/4:.n..d<95.W.
00000B10 BB 0A 29 47 1B CB 9F C7 AB 7F 2E 33 E4 C4 02 88 ..)G.......3....
00000B20 FA 5D 3F C5 F5 48 E2 34 0A 44 E0 75 E7 D0 D5 EE .]?..H.4.D.u....
00000B30 6C 0F 05 E1 4C 4D D7 EA E3 23 D0 77 84 8B 5D 36 l...LM...#.w..]6
00000B40 DA C8 9F CD E4 9B 7C 82 F5 1A ED 97 BD C3 C4 63 ......|........c
00000B50 A8 1F DD 1D 1E 7D DC 79 26 6D CD AB 13 39 BB 77 .....}.y&m...9.w
00000B60 25 D0 03 38 44 FD 0A 1C 7F C7 ED 07 2B 8C 2C CC %..8D.......+.,.
00000B70 DC B4 BC 01 0D 2B F4 9C CF 4D 6D 00 3E C5 4F 79 .....+...Mm.>.Oy
00000B80 AD 63 BB BC 21 8D AD B7 AD 32 6A 83 19 10 8A 7F .c..!....2j.....
00000B90 7F A2 FD 55 AF E6 F3 8F A7 F7 57 15 44 73 ED 1A ...U......W.Ds..
00000BA0 AA 3F 51 F6 8A 14 F9 6E AC 81 8C 19 5A 52 9E 15 .?Q....n....ZR..
00000BB0 B1 1A 0C 69 B8 8F 3B EB 19 D4 53 89 6D 4D 20 71 ...i..;...S.mM q
00000BC0 6D 40 4D 6D 45 76 70 75 50 6B 09 DB AE 86 AE F4 m@MmEvpuPk......
00000BD0 E4 DF 7B F5 17 C0 AA 0E 61 A5 EB 68 F9 73 E3 9B ..{.....a..h.s..
00000BE0 C9 BB 3E 99 85 91 7C 70 D1 4A 7D 91 F9 28 F4 2C ..>...|p.J}..(.,
00000BF0 EF C2 87 D2 D5 98 A2 7A 13 E4 BF AF AD DD 10 98 .......z........


Thanks.

Wait a minute!! That luksDump information that you posted in #4 does not agree with what you had in your original post. There are subtle differences in the MK Digest, MK Salt, and Key Slot 0 Salt. I assume that what is on the disk now is correct and that you made errors in hand-copying data into your posts. None of that would affect whether the LUKS header would be seen as valid, but would definitely prevent a passphrase from being validated.

Hi rknichols

it seems to be in real time :-)

If can help...

Dec 9 11:47:14 Nebula kernel: [587634.613849] sdb: sdb1
Dec 9 11:47:14 Nebula kernel: [587634.615265] sdb: sdb1
Dec 9 11:47:14 Nebula kernel: [587634.617119] sdc: sdc1
Dec 9 11:47:14 Nebula kernel: [587634.619230] sdc: sdc1
Dec 9 11:47:14 Nebula kernel: [587634.658338] sdd: sdd1
Dec 9 11:47:14 Nebula kernel: [587634.696364] sdd: sdd1
Dec 9 11:47:14 Nebula systemd-udevd[27915]: Process '/sbin/mdadm -If sdb1 --path pci-0000:00:1f.2-ata-2' failed with exit code 1.
Dec 9 11:47:14 Nebula kernel: [587634.712055] sdf: sdf1
Dec 9 11:47:14 Nebula kernel: [587634.726599] sdf: sdf1
Dec 9 11:47:14 Nebula kernel: [587634.755650] sde: sde1
Dec 9 11:47:14 Nebula kernel: [587634.784681] sde: sde1
Dec 9 11:47:14 Nebula kernel: [587634.799508] sdg: sdg1
Dec 9 11:47:14 Nebula kernel: [587634.814379] sdg: sdg1
Dec 9 11:47:14 Nebula kernel: [587634.845751] sdh: sdh1
Dec 9 11:47:14 Nebula kernel: [587634.877834] sdh: sdh1
Dec 9 11:47:14 Nebula kernel: [587634.921205] sdi: sdi1
Dec 9 11:47:14 Nebula kernel: [587634.963246] sdi: sdi1
Dec 9 11:47:15 Nebula kernel: [587634.977617] sdk: sdk1
Dec 9 11:47:15 Nebula kernel: [587634.991728] sdk: sdk1
Dec 9 11:47:15 Nebula kernel: [587635.029858] sdj: sdj1
Dec 9 11:47:15 Nebula kernel: [587635.066923] sdj: sdj1
Dec 9 11:47:15 Nebula systemd-udevd[27916]: Process '/sbin/mdadm -If sdc1 --path pci-0000:00:1f.5-ata-1' failed with exit code 1.
Dec 9 11:47:15 Nebula systemd-udevd[27932]: Process '/sbin/mdadm -If sdd1 --path pci-0000:07:00.0-sas-phy0-lun-0' failed with exit code 1.
Dec 9 11:47:15 Nebula systemd-udevd[27951]: Process '/sbin/mdadm -If sde1 --path pci-0000:07:00.0-sas-phy1-lun-0' failed with exit code 1.
Dec 9 11:47:15 Nebula systemd-udevd[27950]: Process '/sbin/mdadm -If sdf1 --path pci-0000:07:00.0-sas-phy2-lun-0' failed with exit code 1.
Dec 9 11:47:15 Nebula systemd-udevd[27958]: Process '/sbin/mdadm -If sdi1 --path pci-0000:07:00.0-sas-phy5-lun-0' failed with exit code 1.
Dec 9 11:47:15 Nebula systemd-udevd[27950]: Process '/sbin/mdadm -If sdg1 --path pci-0000:07:00.0-sas-phy3-lun-0' failed with exit code 1.
Dec 9 11:47:15 Nebula systemd-udevd[27956]: Process '/sbin/mdadm -If sdk1 --path pci-0000:07:00.0-sas-phy7-lun-0' failed with exit code 1.
Dec 9 11:47:15 Nebula systemd-udevd[27932]: Process '/sbin/mdadm -If sdj1 --path pci-0000:07:00.0-sas-phy6-lun-0' failed with exit code 1.
Dec 9 11:47:15 Nebula systemd-udevd[27916]: Process '/sbin/mdadm -If sdh1 --path pci-0000:07:00.0-sas-phy4-lun-0' failed with exit code 1.
Dec 9 11:47:15 Nebula kernel: [587635.124550] md: bind<sdc1>
Dec 9 11:47:15 Nebula kernel: [587635.128911] md: bind<sdb1>
Dec 9 11:47:15 Nebula kernel: [587635.180715] md: bind<sde1>
Dec 9 11:47:15 Nebula kernel: [587635.184390] md: bind<sdd1>
Dec 9 11:47:15 Nebula kernel: [587635.187903] md: bind<sdi1>
Dec 9 11:47:15 Nebula kernel: [587635.191854] md: bind<sdf1>
Dec 9 11:47:15 Nebula kernel: [587635.195263] md: bind<sdh1>
Dec 9 11:47:15 Nebula systemd-udevd[27955]: Process '/sbin/mdadm -If sdc1 --path pci-0000:00:1f.5-ata-1' failed with exit code 1.
Dec 9 11:47:15 Nebula systemd-udevd[27917]: Process '/sbin/mdadm -If sdb1 --path pci-0000:00:1f.2-ata-2' failed with exit code 1.
Dec 9 11:47:15 Nebula kernel: [587635.199707] md: bind<sdg1>
Dec 9 11:47:15 Nebula kernel: [587635.205168] md: bind<sdj1>
Dec 9 11:47:15 Nebula kernel: [587635.213080] md: bind<sdk1>
Dec 9 11:47:15 Nebula systemd-udevd[27955]: Process '/sbin/mdadm --incremental /dev/sdc1 --offroot' failed with exit code 1.
Dec 9 11:47:15 Nebula kernel: [587635.234156] md/raid:md0: device sdk1 operational as raid disk 2
Dec 9 11:47:15 Nebula kernel: [587635.234160] md/raid:md0: device sdj1 operational as raid disk 6
Dec 9 11:47:15 Nebula kernel: [587635.234163] md/raid:md0: device sdh1 operational as raid disk 4
Dec 9 11:47:15 Nebula kernel: [587635.234165] md/raid:md0: device sdf1 operational as raid disk 7
Dec 9 11:47:15 Nebula kernel: [587635.234167] md/raid:md0: device sdi1 operational as raid disk 5
Dec 9 11:47:15 Nebula kernel: [587635.234169] md/raid:md0: device sdd1 operational as raid disk 1
Dec 9 11:47:15 Nebula kernel: [587635.234171] md/raid:md0: device sde1 operational as raid disk 3
Dec 9 11:47:15 Nebula kernel: [587635.234172] md/raid:md0: device sdc1 operational as raid disk 0
Dec 9 11:47:15 Nebula kernel: [587635.235062] md/raid:md0: allocated 8606kB
Dec 9 11:47:15 Nebula kernel: [587635.235100] md/raid:md0: raid level 6 active with 8 out of 8 devices, algorithm 2
Dec 9 11:47:15 Nebula kernel: [587635.235102] RAID conf printout:
Dec 9 11:47:15 Nebula kernel: [587635.235103] --- level:6 rd:8 wd:8
Dec 9 11:47:15 Nebula kernel: [587635.235105] disk 0, o:1, dev:sdc1
Dec 9 11:47:15 Nebula kernel: [587635.235107] disk 1, o:1, dev:sdd1
Dec 9 11:47:15 Nebula kernel: [587635.235109] disk 2, o:1, dev:sdk1
Dec 9 11:47:15 Nebula kernel: [587635.235110] disk 3, o:1, dev:sde1
Dec 9 11:47:15 Nebula kernel: [587635.235112] disk 4, o:1, dev:sdh1
Dec 9 11:47:15 Nebula kernel: [587635.235114] disk 5, o:1, dev:sdi1
Dec 9 11:47:15 Nebula kernel: [587635.235115] disk 6, o:1, dev:sdj1
Dec 9 11:47:15 Nebula kernel: [587635.235117] disk 7, o:1, dev:sdf1
Dec 9 11:47:15 Nebula kernel: [587635.235905] created bitmap (8 pages) for device md0
Dec 9 11:47:15 Nebula kernel: [587635.236261] md0: bitmap initialized from disk: read 1 pages, set 0 of 14900 bits
Dec 9 11:47:15 Nebula systemd-udevd[27917]: Process '/sbin/mdadm --incremental /dev/sdb1 --offroot' failed with exit code 1.
Dec 9 11:47:15 Nebula mdadm[2314]: NewArray event detected on md device /dev/md0
Dec 9 11:47:15 Nebula kernel: [587635.251215] md0: detected capacity change from 0 to 11999014551552
Dec 9 11:47:15 Nebula kernel: [587635.251276] RAID conf printout:
Dec 9 11:47:15 Nebula kernel: [587635.251279] --- level:6 rd:8 wd:8
Dec 9 11:47:15 Nebula kernel: [587635.251281] disk 0, o:1, dev:sdc1
Dec 9 11:47:15 Nebula kernel: [587635.251284] disk 1, o:1, dev:sdd1
Dec 9 11:47:15 Nebula kernel: [587635.251286] disk 2, o:1, dev:sdk1
Dec 9 11:47:15 Nebula kernel: [587635.251288] disk 3, o:1, dev:sde1
Dec 9 11:47:15 Nebula kernel: [587635.251290] disk 4, o:1, dev:sdh1
Dec 9 11:47:15 Nebula kernel: [587635.251292] disk 5, o:1, dev:sdi1
Dec 9 11:47:15 Nebula kernel: [587635.251295] disk 6, o:1, dev:sdj1
Dec 9 11:47:15 Nebula kernel: [587635.251296] disk 7, o:1, dev:sdf1
Dec 9 11:47:15 Nebula kernel: [587635.251298] RAID conf printout:
Dec 9 11:47:15 Nebula kernel: [587635.251299] --- level:6 rd:8 wd:8
Dec 9 11:47:15 Nebula kernel: [587635.251300] disk 0, o:1, dev:sdc1
Dec 9 11:47:15 Nebula kernel: [587635.251301] disk 1, o:1, dev:sdd1
Dec 9 11:47:15 Nebula kernel: [587635.251302] disk 2, o:1, dev:sdk1
Dec 9 11:47:15 Nebula kernel: [587635.251303] disk 3, o:1, dev:sde1
Dec 9 11:47:15 Nebula kernel: [587635.251305] disk 4, o:1, dev:sdh1
Dec 9 11:47:15 Nebula kernel: [587635.251306] disk 5, o:1, dev:sdi1
Dec 9 11:47:15 Nebula kernel: [587635.251307] disk 6, o:1, dev:sdj1
Dec 9 11:47:15 Nebula kernel: [587635.251308] disk 7, o:1, dev:sdf1
Dec 9 11:47:15 Nebula kernel: [587635.264552] md: md0 still in use.
Dec 9 11:47:15 Nebula kernel: [587635.264983] md/raid:md0: Disk failure on sde1, disabling device.
Dec 9 11:47:15 Nebula kernel: [587635.264983] md/raid:md0: Operation continuing on 7 devices.
Dec 9 11:47:15 Nebula kernel: [587635.304773] RAID conf printout:
Dec 9 11:47:15 Nebula kernel: [587635.304778] --- level:6 rd:8 wd:7
Dec 9 11:47:15 Nebula kernel: [587635.304780] disk 0, o:1, dev:sdc1
Dec 9 11:47:15 Nebula kernel: [587635.304782] disk 1, o:1, dev:sdd1
Dec 9 11:47:15 Nebula kernel: [587635.304784] disk 2, o:1, dev:sdk1
Dec 9 11:47:15 Nebula kernel: [587635.304785] disk 3, o:0, dev:sde1
Dec 9 11:47:15 Nebula kernel: [587635.304787] disk 4, o:1, dev:sdh1
Dec 9 11:47:15 Nebula kernel: [587635.304789] disk 5, o:1, dev:sdi1
Dec 9 11:47:15 Nebula kernel: [587635.304790] disk 6, o:1, dev:sdj1
Dec 9 11:47:15 Nebula kernel: [587635.304792] disk 7, o:1, dev:sdf1
Dec 9 11:47:15 Nebula mdadm[2314]: Fail event detected on md device /dev/md0, component device /dev/sde1
Dec 9 11:47:15 Nebula postfix/pickup[17208]: 538171D82DF8: uid=0 from=<root>
Dec 9 11:47:15 Nebula postfix/cleanup[28205]: 538171D82DF8: message-id=<20171209104715.538171D82DF8@nebula.localdomain>
Dec 9 11:47:15 Nebula mdadm[2314]: RebuildStarted event detected on md device /dev/md0
Dec 9 11:47:15 Nebula kernel: [587635.327481] RAID conf printout:
Dec 9 11:47:15 Nebula kernel: [587635.327486] --- level:6 rd:8 wd:7
Dec 9 11:47:15 Nebula kernel: [587635.327489] disk 0, o:1, dev:sdc1
Dec 9 11:47:15 Nebula kernel: [587635.327491] disk 1, o:1, dev:sdd1
Dec 9 11:47:15 Nebula kernel: [587635.327494] disk 2, o:1, dev:sdk1
Dec 9 11:47:15 Nebula kernel: [587635.327496] disk 4, o:1, dev:sdh1
Dec 9 11:47:15 Nebula kernel: [587635.327498] disk 5, o:1, dev:sdi1
Dec 9 11:47:15 Nebula kernel: [587635.327500] disk 6, o:1, dev:sdj1
Dec 9 11:47:15 Nebula kernel: [587635.327502] disk 7, o:1, dev:sdf1

Dec 9 11:45:27 Nebula cron[1854]: postdrop: warning: uid=1000: File too large
Dec 9 11:45:27 Nebula postfix/postdrop[21597]: warning: uid=1000: File too large
Dec 9 11:45:27 Nebula cron[1854]: sendmail: fatal: adriano(1000): message file too big
Dec 9 11:45:27 Nebula postfix/sendmail[21596]: fatal: adriano(1000): message file too big
Dec 9 11:45:27 Nebula CRON[14067]: (adriano) MAIL (mailed 33685007 bytes of output but got status 0x004b from MTA#012)

That is clearly not the first few sectors of the LUKS header. It appears to be encrypted data from somewhere within the container. The md array is not properly assembled. You have to get it assembled such that the sectors that you recovered from /dev/sdd1 in post #4 appear at the start of the assembled array. I'm afraid I'm not much of an expert on mdadm RAID 6, so I can't help you much there.

And now I see that you have a failed disk and a rebuild going on with an array that is known to be improperly assembled. All I can say at this point is that I hope you have good backups because there is a very good chance that you have just lost the whole thing.

Hi rknichols

Sorry for the silence... but after your last message I had to suspend the recovery for lack of time,
this things can not be done in a hurry!

I had the backup of almost all of the RAID but I really wanted to try to recover it,
both in order to lose nothing and to learn something more.

I wanted to thank you for your help!
You put me on the right track to solve the problem!

In fact, the problem was related to the assembly of the RAID that was wrong.
With a little patience and a lot of googling I managed to solve the problem by recovering everything.

I found help on these sites:

https://raid.wiki.kernel.org/index.p...hings_Go_Wrogn
https://www.linuxquestions.org/quest...se-4175538773/

I put the operations I have done below, hoping they can help someone else.

First I save the post mortem status of the RAID:

mdadm --misc --examine --verbose /dev/sd[bcdefghijk]1 > raid.status.original

/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 1d8cd572:a5e57fa7:3b0eec8b:abe08b5b
Name : Nebula:0
Creation Time : Wed Mar 16 23:21:21 2011
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905929222 (1862.49 GiB 1999.84 GB)
Array Size : 11717787648 (11174.95 GiB 11999.01 GB)
Used Dev Size : 3905929216 (1862.49 GiB 1999.84 GB)
Data Offset : 376 sectors
Super Offset : 8 sectors
Unused Space : before=296 sectors, after=6 sectors
State : clean
Device UUID : 205ca756:5fa034a5:29e48699:3dfe8ff3

Internal Bitmap : 8 sectors from superblock
Update Time : Sat Dec 9 11:47:16 2017
Checksum : 521f1d26 - correct
Events : 543320

Layout : left-symmetric
Chunk Size : 512K

Device Role : spare
Array State : A....... ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 1d8cd572:a5e57fa7:3b0eec8b:abe08b5b
Name : Nebula:0
Creation Time : Wed Mar 16 23:21:21 2011
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905929222 (1862.49 GiB 1999.84 GB)
Array Size : 11717787648 (11174.95 GiB 11999.01 GB)
Used Dev Size : 3905929216 (1862.49 GiB 1999.84 GB)
Data Offset : 376 sectors
Super Offset : 8 sectors
Unused Space : before=296 sectors, after=6 sectors
State : clean
Device UUID : cd99db50:1715cd16:53712b2a:f087797e

Internal Bitmap : 8 sectors from superblock
Update Time : Sat Dec 9 11:47:16 2017
Checksum : d979e667 - correct
Events : 543320

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 0
Array State : A....... ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 1d8cd572:a5e57fa7:3b0eec8b:abe08b5b
Name : Nebula:0
Creation Time : Wed Mar 16 23:21:21 2011
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905929326 (1862.49 GiB 1999.84 GB)
Array Size : 11717787648 (11174.95 GiB 11999.01 GB)
Used Dev Size : 3905929216 (1862.49 GiB 1999.84 GB)
Data Offset : 272 sectors
Super Offset : 8 sectors
Unused Space : before=192 sectors, after=110 sectors
State : clean
Device UUID : c7a8a7bf:76e1521d:6077d914:15b5b45b

Internal Bitmap : 8 sectors from superblock
Update Time : Sat Dec 9 11:47:15 2017
Checksum : 16c4f4f7 - correct
Events : 543308

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 1
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 1d8cd572:a5e57fa7:3b0eec8b:abe08b5b
Name : Nebula:0
Creation Time : Wed Mar 16 23:21:21 2011
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905929222 (1862.49 GiB 1999.84 GB)
Array Size : 11717787648 (11174.95 GiB 11999.01 GB)
Used Dev Size : 3905929216 (1862.49 GiB 1999.84 GB)
Data Offset : 376 sectors
Super Offset : 8 sectors
Unused Space : before=296 sectors, after=6 sectors
State : clean
Device UUID : 825cace3:a4e0a2d9:6fe43fe7:fe256632

Internal Bitmap : 8 sectors from superblock
Update Time : Sat Dec 9 11:45:50 2017
Checksum : a02c8583 - correct
Events : 543304

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 3
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 1d8cd572:a5e57fa7:3b0eec8b:abe08b5b
Name : Nebula:0
Creation Time : Wed Mar 16 23:21:21 2011
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905929222 (1862.49 GiB 1999.84 GB)
Array Size : 11717787648 (11174.95 GiB 11999.01 GB)
Used Dev Size : 3905929216 (1862.49 GiB 1999.84 GB)
Data Offset : 376 sectors
Super Offset : 8 sectors
Unused Space : before=296 sectors, after=6 sectors
State : clean
Device UUID : 646b6501:b70d135f:18e2898c:f3f83be7
Internal Bitmap : 8 sectors from superblock
Update Time : Sat Dec 9 11:47:15 2017
Checksum : 9d7a926a - correct
Events : 543308

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 7
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x3
Array UUID : 1d8cd572:a5e57fa7:3b0eec8b:abe08b5b
Name : Nebula:0
Creation Time : Wed Mar 16 23:21:21 2011
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905929222 (1862.49 GiB 1999.84 GB)
Array Size : 11717787648 (11174.95 GiB 11999.01 GB)
Used Dev Size : 3905929216 (1862.49 GiB 1999.84 GB)
Data Offset : 376 sectors
Super Offset : 8 sectors
Recovery Offset : 0 sectors
Unused Space : before=296 sectors, after=6 sectors
State : clean
Device UUID : 70386324:950e6081:985bd3bd:2aaa2531

Internal Bitmap : 8 sectors from superblock
Update Time : Sat Dec 9 11:47:15 2017
Checksum : 5df88b0c - correct
Events : 543308

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 3
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdh1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 1d8cd572:a5e57fa7:3b0eec8b:abe08b5b
Name : Nebula:0
Creation Time : Wed Mar 16 23:21:21 2011
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905929326 (1862.49 GiB 1999.84 GB)
Array Size : 11717787648 (11174.95 GiB 11999.01 GB)
Used Dev Size : 3905929216 (1862.49 GiB 1999.84 GB)
Data Offset : 272 sectors
Super Offset : 8 sectors
Unused Space : before=192 sectors, after=110 sectors
State : clean
Device UUID : e1a37958:0af549bf:2f759ca1:d442b3c7

Internal Bitmap : 8 sectors from superblock
Update Time : Sat Dec 9 11:47:15 2017
Checksum : 4a418f2e - correct
Events : 543313

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 4
Array State : A.A.A.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdi1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 1d8cd572:a5e57fa7:3b0eec8b:abe08b5b
Name : Nebula:0
Creation Time : Wed Mar 16 23:21:21 2011
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905929326 (1862.49 GiB 1999.84 GB)
Array Size : 11717787648 (11174.95 GiB 11999.01 GB)
Used Dev Size : 3905929216 (1862.49 GiB 1999.84 GB)
Data Offset : 272 sectors
Super Offset : 8 sectors
Unused Space : before=192 sectors, after=110 sectors
State : clean
Device UUID : f8be7997:619bbf14:ace786a3:08a6e072

Internal Bitmap : 8 sectors from superblock
Update Time : Sat Dec 9 11:47:15 2017
Checksum : 8bdd264a - correct
Events : 543308

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 5
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdj1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 1d8cd572:a5e57fa7:3b0eec8b:abe08b5b
Name : Nebula:0
Creation Time : Wed Mar 16 23:21:21 2011
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905929326 (1862.49 GiB 1999.84 GB)
Array Size : 11717787648 (11174.95 GiB 11999.01 GB)
Used Dev Size : 3905929216 (1862.49 GiB 1999.84 GB)
Data Offset : 272 sectors
Super Offset : 8 sectors
Unused Space : before=192 sectors, after=110 sectors
State : clean
Device UUID : ab0f4811:ab371a2c:890f9763:8b9e5b12

Internal Bitmap : 8 sectors from superblock
Update Time : Sat Dec 9 11:47:15 2017
Checksum : 7c8433a2 - correct
Events : 543315

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 6
Array State : A.A...A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdk1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 1d8cd572:a5e57fa7:3b0eec8b:abe08b5b
Name : Nebula:0
Creation Time : Wed Mar 16 23:21:21 2011
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905929222 (1862.49 GiB 1999.84 GB)
Array Size : 11717787648 (11174.95 GiB 11999.01 GB)
Used Dev Size : 3905929216 (1862.49 GiB 1999.84 GB)
Data Offset : 376 sectors
Super Offset : 8 sectors
Unused Space : before=296 sectors, after=6 sectors
State : clean
Device UUID : 12b23617:d570950a:6d42a561:61de31bd

Internal Bitmap : 8 sectors from superblock
Update Time : Sat Dec 9 11:47:15 2017
Checksum : 9d281f1 - correct
Events : 543315

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 2
Array State : A.A...A. ('A' == active, '.' == missing, 'R' == replacing)

and check in the old log the correct sequence of the drives on the RAID before the fail.

Dec 1 20:00:14 Nebula kernel: [ 12.669080] md: raid6 personality registered for level 6
Dec 1 20:00:14 Nebula kernel: [ 12.669083] md: raid5 personality registered for level 5
Dec 1 20:00:14 Nebula kernel: [ 12.669084] md: raid4 personality registered for level 4
Dec 1 20:00:14 Nebula kernel: [ 12.671093] md/raid:md0: device sdj1 operational as raid disk 6
Dec 1 20:00:14 Nebula kernel: [ 12.671096] md/raid:md0: device sdi1 operational as raid disk 5
Dec 1 20:00:14 Nebula kernel: [ 12.671097] md/raid:md0: device sdh1 operational as raid disk 4
Dec 1 20:00:14 Nebula kernel: [ 12.671098] md/raid:md0: device sde1 operational as raid disk 3
Dec 1 20:00:14 Nebula kernel: [ 12.671099] md/raid:md0: device sdk1 operational as raid disk 2
Dec 1 20:00:14 Nebula kernel: [ 12.671101] md/raid:md0: device sdf1 operational as raid disk 7
Dec 1 20:00:14 Nebula kernel: [ 12.671102] md/raid:md0: device sdd1 operational as raid disk 1
Dec 1 20:00:14 Nebula kernel: [ 12.671103] md/raid:md0: device sdc1 operational as raid disk 0
Dec 1 20:00:14 Nebula kernel: [ 12.671664] md/raid:md0: allocated 8606kB
Dec 1 20:00:14 Nebula kernel: [ 12.671783] md/raid:md0: raid level 6 active with 8 out of 8 devices, algorithm 2
Dec 1 20:00:14 Nebula kernel: [ 12.671786] RAID conf printout:
Dec 1 20:00:14 Nebula kernel: [ 12.671787] --- level:6 rd:8 wd:8
Dec 1 20:00:14 Nebula kernel: [ 12.671789] disk 0, o:1, dev:sdc1
Dec 1 20:00:14 Nebula kernel: [ 12.671791] disk 1, o:1, dev:sdd1
Dec 1 20:00:14 Nebula kernel: [ 12.671793] disk 2, o:1, dev:sdk1
Dec 1 20:00:14 Nebula kernel: [ 12.671795] disk 3, o:1, dev:sde1
Dec 1 20:00:14 Nebula kernel: [ 12.671796] disk 4, o:1, dev:sdh1
Dec 1 20:00:14 Nebula kernel: [ 12.671798] disk 5, o:1, dev:sdi1
Dec 1 20:00:14 Nebula kernel: [ 12.671800] disk 6, o:1, dev:sdj1
Dec 1 20:00:14 Nebula kernel: [ 12.671802] disk 7, o:1, dev:sdf1
Dec 1 20:00:14 Nebula kernel: [ 12.672007] created bitmap (8 pages) for device md0
Dec 1 20:00:14 Nebula kernel: [ 12.672383] md0: bitmap initialized from disk: read 1 pages, set 0 of 14900 bits
Dec 1 20:00:14 Nebula kernel: [ 12.710331] md0: detected capacity change from 0 to 11999014551552

So I sort the Events for every drives in order to understand the fault sequence:

PM OK
role role Status

/dev/sde1: 3 3 Events : 543304 AAAAAAAA

/dev/sdd1: 1 1 Events : 543308 AAAAAAAA
/dev/sdf1: 7 7 Events : 543308 AAAAAAAA
/dev/sdg1: 3 - Events : 543308 AAAAAAAA
/dev/sdi1: 5 5 Events : 543308 AAAAAAAA

/dev/sdh1: 4 4 Events : 543313 A.A.A.A.

/dev/sdj1: 6 6 Events : 543315 A.A...A.
/dev/sdk1: 2 2 Events : 543315 A.A...A.

/dev/sdb1: 8 - Events : 543320 A....... spare
/dev/sdc1: 0 0 Events : 543320 A.......

After I decide to try to recreate the RAID header

mdadm --create /dev/md0 --metadata=1.2 --assume-clean --bitmap=internal --bitmap-chunk=131072 --chunk=512 --level=6 --raid-devices=8 --verbose /dev/sd[cdkehijf]1

but this command give me the bad sequence!
so I tried with this:

# mdadm --create /dev/md0 --metadata=1.2 --assume-clean --bitmap=internal --bitmap-chunk=131072 --chunk=512 --level=6 --raid-devices=8 --verbose /dev/sdc1 /dev/sdd1 /dev/sdk1 /dev/sde1 /dev/sdh1 /dev/sdi1 /dev/sdj1 /dev/sdf1

mdadm: array /dev/md0 started.

# cat /proc/mdstat

Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md0 : active raid6 sdf1[7] sdj1[6] sdi1[5] sdh1[4] sde1[3] sdk1[2] sdd1[1] sdc1[0]
11717001216 blocks super 1.2 level 6, 512k chunk, algorithm 2 [8/8] [UUUUUUUU]
bitmap: 0/8 pages [0KB], 131072KB chunk

# cryptsetup -v isLuks /dev/md0
Device /dev/md0 is not a valid LUKS device.
Command failed with code 22: Device /dev/md0 is not a valid LUKS device.

No good! ... I take a break :-( and analyse again the RAID...........

Best regards

Adriano

Hi rknichols

Never give up! 2nd round...

mdadm --misc --examine --verbose /dev/sd[bcdefghijk]1 > raid.status.new

/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 1d8cd572:a5e57fa7:3b0eec8b:abe08b5b
Name : Nebula:0
Creation Time : Wed Mar 16 23:21:21 2011
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905929222 (1862.49 GiB 1999.84 GB)
Array Size : 11717787648 (11174.95 GiB 11999.01 GB)
Used Dev Size : 3905929216 (1862.49 GiB 1999.84 GB)
Data Offset : 376 sectors
Super Offset : 8 sectors
Unused Space : before=296 sectors, after=6 sectors
State : clean
Device UUID : 205ca756:5fa034a5:29e48699:3dfe8ff3

Internal Bitmap : 8 sectors from superblock
Update Time : Sat Dec 9 11:47:16 2017
Checksum : 521f1d26 - correct
Events : 543320

Layout : left-symmetric
Chunk Size : 512K

Device Role : spare
Array State : A....... ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 57a12a53:5b882e65:28002347:cfd37d69
Name : nebula:0 (local to host nebula)
Creation Time : Mon Dec 11 00:14:59 2017
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905667454 (1862.37 GiB 1999.70 GB)
Array Size : 11717001216 (11174.20 GiB 11998.21 GB)
Used Dev Size : 3905667072 (1862.37 GiB 1999.70 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=382 sectors
State : clean
Device UUID : b45fa8b6:56f996b1:508a9c3e:3afc5eee

Internal Bitmap : 8 sectors from superblock
Update Time : Mon Dec 11 00:14:59 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : d30bf974 - correct
Events : 0

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 0
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 57a12a53:5b882e65:28002347:cfd37d69
Name : nebula:0 (local to host nebula)
Creation Time : Mon Dec 11 00:14:59 2017
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905667454 (1862.37 GiB 1999.70 GB)
Array Size : 11717001216 (11174.20 GiB 11998.21 GB)
Used Dev Size : 3905667072 (1862.37 GiB 1999.70 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=382 sectors
State : clean
Device UUID : bcdf9030:f4b93ea4:ee4c6c0a:bbf862cc

Internal Bitmap : 8 sectors from superblock
Update Time : Mon Dec 11 00:14:59 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : e96ff939 - correct
Events : 0

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 1
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 57a12a53:5b882e65:28002347:cfd37d69
Name : nebula:0 (local to host nebula)
Creation Time : Mon Dec 11 00:14:59 2017
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905667454 (1862.37 GiB 1999.70 GB)
Array Size : 11717001216 (11174.20 GiB 11998.21 GB)
Used Dev Size : 3905667072 (1862.37 GiB 1999.70 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=382 sectors
State : clean
Device UUID : 93fee86d:b2bd75d1:db5c176f:0c039c4a

Internal Bitmap : 8 sectors from superblock
Update Time : Mon Dec 11 00:14:59 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 36e3360f - correct
Events : 0

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 3
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 57a12a53:5b882e65:28002347:cfd37d69
Name : nebula:0 (local to host nebula)
Creation Time : Mon Dec 11 00:14:59 2017
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905667454 (1862.37 GiB 1999.70 GB)
Array Size : 11717001216 (11174.20 GiB 11998.21 GB)
Used Dev Size : 3905667072 (1862.37 GiB 1999.70 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=382 sectors
State : clean
Device UUID : e777b70f:021da66c:6529fb9b:f996c410

Internal Bitmap : 8 sectors from superblock
Update Time : Mon Dec 11 00:14:59 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 66ee6f2d - correct
Events : 0

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 7
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : dfa6ea79:3f1f621e:4df96e54:5b65f99c
Name : nebula:0 (local to host nebula)
Creation Time : Sun Dec 10 22:45:25 2017
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905667454 (1862.37 GiB 1999.70 GB)
Array Size : 11717001216 (11174.20 GiB 11998.21 GB)
Used Dev Size : 3905667072 (1862.37 GiB 1999.70 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=382 sectors
State : clean
Device UUID : 1fd38ace:b320b696:9ad6c8a3:31ae53dc

Internal Bitmap : 8 sectors from superblock
Update Time : Sun Dec 10 22:45:25 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 43e98fa5 - correct
Events : 0

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 6
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdh1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 57a12a53:5b882e65:28002347:cfd37d69
Name : nebula:0 (local to host nebula)
Creation Time : Mon Dec 11 00:14:59 2017
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905667454 (1862.37 GiB 1999.70 GB)
Array Size : 11717001216 (11174.20 GiB 11998.21 GB)
Used Dev Size : 3905667072 (1862.37 GiB 1999.70 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=382 sectors
State : clean
Device UUID : 92c04513:ced60444:43120c09:09daf051

Internal Bitmap : 8 sectors from superblock
Update Time : Mon Dec 11 00:14:59 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : f0189d8e - correct
Events : 0

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 4
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdi1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 57a12a53:5b882e65:28002347:cfd37d69
Name : nebula:0 (local to host nebula)
Creation Time : Mon Dec 11 00:14:59 2017
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905667454 (1862.37 GiB 1999.70 GB)
Array Size : 11717001216 (11174.20 GiB 11998.21 GB)
Used Dev Size : 3905667072 (1862.37 GiB 1999.70 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=382 sectors
State : clean
Device UUID : b8c5ab88:f2951115:6d95296c:7b00d0bc

Internal Bitmap : 8 sectors from superblock
Update Time : Mon Dec 11 00:14:59 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 4880b77 - correct
Events : 0

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 5
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdj1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 57a12a53:5b882e65:28002347:cfd37d69
Name : nebula:0 (local to host nebula)
Creation Time : Mon Dec 11 00:14:59 2017
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905667454 (1862.37 GiB 1999.70 GB)
Array Size : 11717001216 (11174.20 GiB 11998.21 GB)
Used Dev Size : 3905667072 (1862.37 GiB 1999.70 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=382 sectors
State : clean
Device UUID : 5616a3d7:621493ec:fffec018:707bb48c

Internal Bitmap : 8 sectors from superblock
Update Time : Mon Dec 11 00:14:59 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : a77cbf0d - correct
Events : 0

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 6
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdk1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 57a12a53:5b882e65:28002347:cfd37d69
Name : nebula:0 (local to host nebula)
Creation Time : Mon Dec 11 00:14:59 2017
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905667454 (1862.37 GiB 1999.70 GB)
Array Size : 11717001216 (11174.20 GiB 11998.21 GB)
Used Dev Size : 3905667072 (1862.37 GiB 1999.70 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=382 sectors
State : clean
Device UUID : 5aed04eb:3ace1012:0454159d:5fe45927

Internal Bitmap : 8 sectors from superblock
Update Time : Mon Dec 11 00:14:59 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : ff560dd8 - correct
Events : 0

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 2
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)


So I notice than the "Data Offset" was wrong.
After hours of search on google I found that it was possible to indicate the specific offset for each drive in the array.
Yes, because as you can see the offsets are not the same for all drives, in my case they vary depending on the physical size of the HD (2TB 272s / 4TB 376s ...)

So I try again!

# mdadm --create /dev/md0 --metadata=1.2 --assume-clean --auto=md --bitmap=internal --bitmap-chunk=131072 --chunk=512 --data-offset=variable --level=6 --raid-devices=8 --verbose /dev/sdc1:376s /dev/sdd1:272s /dev/sdk1:376s /dev/sde1:376s /dev/sdh1:272s /dev/sdi1:272s /dev/sdj1:272s /dev/sdf1:376s

mdadm: layout defaults to left-symmetric
mdadm: /dev/sdc1 appears to be part of a raid array:
level=raid6 devices=8 ctime=Mon Dec 11 02:10:47 2017
mdadm: /dev/sdd1 appears to be part of a raid array:
level=raid6 devices=8 ctime=Mon Dec 11 02:10:47 2017
mdadm: /dev/sdk1 appears to be part of a raid array:
level=raid6 devices=8 ctime=Mon Dec 11 02:10:47 2017
mdadm: /dev/sde1 appears to be part of a raid array:
level=raid6 devices=8 ctime=Mon Dec 11 02:10:47 2017
mdadm: /dev/sdh1 appears to be part of a raid array:
level=raid6 devices=8 ctime=Mon Dec 11 02:10:47 2017
mdadm: /dev/sdi1 appears to be part of a raid array:
level=raid6 devices=8 ctime=Mon Dec 11 02:10:47 2017
mdadm: /dev/sdj1 appears to be part of a raid array:
level=raid6 devices=8 ctime=Mon Dec 11 02:10:47 2017
mdadm: /dev/sdf1 appears to be part of a raid array:
level=raid6 devices=8 ctime=Mon Dec 11 02:10:47 2017
mdadm: size set to 1952964608K
Continue creating array? Y
mdadm: array /dev/md0 started.

# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md0 : active raid6 sdf1[7] sdj1[6] sdi1[5] sdh1[4] sde1[3] sdk1[2] sdd1[1] sdc1[0]
11717787648 blocks super 1.2 level 6, 512k chunk, algorithm 2 [8/8] [UUUUUUUU]
bitmap: 0/8 pages [0KB], 131072KB chunk

unused devices: <none>

So I try again!

# cryptsetup -v isLuks /dev/md0
Command successful.


# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Sat Dec 16 02:35:16 2017
Raid Level : raid6
Array Size : 11717787648 (11174.95 GiB 11999.01 GB)
Used Dev Size : 1952964608 (1862.49 GiB 1999.84 GB)
Raid Devices : 8
Total Devices : 8
Persistence : Superblock is persistent

Intent Bitmap : Internal

Update Time : Sat Dec 16 02:35:16 2017
State : clean
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 512K

Name : nebula:0 (local to host nebula)
UUID : 0a1a5539:80f64dec:13cc27b7:8e958486
Events : 0

Number Major Minor RaidDevice State
0 8 33 0 active sync /dev/sdc1
1 8 49 1 active sync /dev/sdd1
2 8 161 2 active sync /dev/sdk1
3 8 65 3 active sync /dev/sde1
4 8 113 4 active sync /dev/sdh1
5 8 129 5 active sync /dev/sdi1
6 8 145 6 active sync /dev/sdj1
7 8 81 7 active sync /dev/sdf1

I save the new RAID status:

# mdadm --misc --examine --verbose /dev/sd[bcdefghijk]1 > raid.status.OK
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 1d8cd572:a5e57fa7:3b0eec8b:abe08b5b
Name : Nebula:0
Creation Time : Wed Mar 16 23:21:21 2011
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905929222 (1862.49 GiB 1999.84 GB)
Array Size : 11717787648 (11174.95 GiB 11999.01 GB)
Used Dev Size : 3905929216 (1862.49 GiB 1999.84 GB)
Data Offset : 376 sectors
Super Offset : 8 sectors
Unused Space : before=296 sectors, after=6 sectors
State : clean
Device UUID : 205ca756:5fa034a5:29e48699:3dfe8ff3

Internal Bitmap : 8 sectors from superblock
Update Time : Sat Dec 9 11:47:16 2017
Checksum : 521f1d26 - correct
Events : 543320

Layout : left-symmetric
Chunk Size : 512K

Device Role : spare
Array State : A....... ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 0a1a5539:80f64dec:13cc27b7:8e958486
Name : nebula:0 (local to host nebula)
Creation Time : Sat Dec 16 02:35:16 2017
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905929222 (1862.49 GiB 1999.84 GB)
Array Size : 11717787648 (11174.95 GiB 11999.01 GB)
Used Dev Size : 3905929216 (1862.49 GiB 1999.84 GB)
Data Offset : 376 sectors
Super Offset : 8 sectors
Unused Space : before=288 sectors, after=6 sectors
State : clean
Device UUID : c79ad4d0:3f28deaf:b5d86729:3e6c5feb

Internal Bitmap : 8 sectors from superblock
Update Time : Sat Dec 16 02:35:16 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : cdb2071e - correct
Events : 0

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 0
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 0a1a5539:80f64dec:13cc27b7:8e958486
Name : nebula:0 (local to host nebula)
Creation Time : Sat Dec 16 02:35:16 2017
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905929326 (1862.49 GiB 1999.84 GB)
Array Size : 11717787648 (11174.95 GiB 11999.01 GB)
Used Dev Size : 3905929216 (1862.49 GiB 1999.84 GB)
Data Offset : 272 sectors
Super Offset : 8 sectors
Unused Space : before=184 sectors, after=110 sectors
State : clean
Device UUID : 142b3fa3:603221e5:de72d9da:6ab2d85a

Internal Bitmap : 8 sectors from superblock
Update Time : Sat Dec 16 02:35:16 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : f64a81e2 - correct
Events : 0

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 1
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 0a1a5539:80f64dec:13cc27b7:8e958486
Name : nebula:0 (local to host nebula)
Creation Time : Sat Dec 16 02:35:16 2017
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905929222 (1862.49 GiB 1999.84 GB)
Array Size : 11717787648 (11174.95 GiB 11999.01 GB)
Used Dev Size : 3905929216 (1862.49 GiB 1999.84 GB)
Data Offset : 376 sectors
Super Offset : 8 sectors
Unused Space : before=288 sectors, after=6 sectors
State : clean
Device UUID : 3eb9c25f:eaf5ba6a:01d465be:4ee9f3b1

Internal Bitmap : 8 sectors from superblock
Update Time : Sat Dec 16 02:35:16 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 730f6b9f - correct
Events : 0

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 3
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 0a1a5539:80f64dec:13cc27b7:8e958486
Name : nebula:0 (local to host nebula)
Creation Time : Sat Dec 16 02:35:16 2017
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905929222 (1862.49 GiB 1999.84 GB)
Array Size : 11717787648 (11174.95 GiB 11999.01 GB)
Used Dev Size : 3905929216 (1862.49 GiB 1999.84 GB)
Data Offset : 376 sectors
Super Offset : 8 sectors
Unused Space : before=288 sectors, after=6 sectors
State : clean
Device UUID : 87166840:b8276b12:5a1b1353:7e59fd08

Internal Bitmap : 8 sectors from superblock
Update Time : Sat Dec 16 02:35:16 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : e71bb241 - correct
Events : 0

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 7
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : dfa6ea79:3f1f621e:4df96e54:5b65f99c
Name : nebula:0 (local to host nebula)
Creation Time : Sun Dec 10 22:45:25 2017
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905667454 (1862.37 GiB 1999.70 GB)
Array Size : 11717001216 (11174.20 GiB 11998.21 GB)
Used Dev Size : 3905667072 (1862.37 GiB 1999.70 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=382 sectors
State : clean
Device UUID : 1fd38ace:b320b696:9ad6c8a3:31ae53dc

Internal Bitmap : 8 sectors from superblock
Update Time : Sun Dec 10 22:45:25 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 43e98fa5 - correct
Events : 0

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 6
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdh1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 0a1a5539:80f64dec:13cc27b7:8e958486
Name : nebula:0 (local to host nebula)
Creation Time : Sat Dec 16 02:35:16 2017
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905929326 (1862.49 GiB 1999.84 GB)
Array Size : 11717787648 (11174.95 GiB 11999.01 GB)
Used Dev Size : 3905929216 (1862.49 GiB 1999.84 GB)
Data Offset : 272 sectors
Super Offset : 8 sectors
Unused Space : before=184 sectors, after=110 sectors
State : clean
Device UUID : 8deb3530:42e1975e:265d8497:16064fdc

Internal Bitmap : 8 sectors from superblock
Update Time : Sat Dec 16 02:35:16 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 3ad92f34 - correct
Events : 0

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 4
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdi1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 0a1a5539:80f64dec:13cc27b7:8e958486
Name : nebula:0 (local to host nebula)
Creation Time : Sat Dec 16 02:35:16 2017
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905929326 (1862.49 GiB 1999.84 GB)
Array Size : 11717787648 (11174.95 GiB 11999.01 GB)
Used Dev Size : 3905929216 (1862.49 GiB 1999.84 GB)
Data Offset : 272 sectors
Super Offset : 8 sectors
Unused Space : before=184 sectors, after=110 sectors
State : clean
Device UUID : ace03c9c:1aaf7bf4:cbe40147:46c602aa

Internal Bitmap : 8 sectors from superblock
Update Time : Sat Dec 16 02:35:16 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : b9f53a01 - correct
Events : 0

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 5
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdj1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 0a1a5539:80f64dec:13cc27b7:8e958486
Name : nebula:0 (local to host nebula)
Creation Time : Sat Dec 16 02:35:16 2017
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905929326 (1862.49 GiB 1999.84 GB)
Array Size : 11717787648 (11174.95 GiB 11999.01 GB)
Used Dev Size : 3905929216 (1862.49 GiB 1999.84 GB)
Data Offset : 272 sectors
Super Offset : 8 sectors
Unused Space : before=184 sectors, after=110 sectors
State : clean
Device UUID : 45d70ab1:c8de21d1:61204e71:039637c7

Internal Bitmap : 8 sectors from superblock
Update Time : Sat Dec 16 02:35:16 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : f2ea6b9c - correct
Events : 0

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 6
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdk1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 0a1a5539:80f64dec:13cc27b7:8e958486
Name : nebula:0 (local to host nebula)
Creation Time : Sat Dec 16 02:35:16 2017
Raid Level : raid6
Raid Devices : 8

Avail Dev Size : 3905929222 (1862.49 GiB 1999.84 GB)
Array Size : 11717787648 (11174.95 GiB 11999.01 GB)
Used Dev Size : 3905929216 (1862.49 GiB 1999.84 GB)
Data Offset : 376 sectors
Super Offset : 8 sectors
Unused Space : before=288 sectors, after=6 sectors
State : clean
Device UUID : 14739dec:d9473894:33547108:401963e4

Internal Bitmap : 8 sectors from superblock
Update Time : Sat Dec 16 02:35:16 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : a5e22787 - correct
Events : 0

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 2
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)

Update the UUID on mdadm.conf:

# cd /etc/mdadm/
# geany mdadm.conf

# definitions of existing MD arrays
ARRAY /dev/md0 level=raid6 num-devices=8 metadata=1.2 UUID=0a1a5539:80f64dec:13cc27b7:8e958486 name=Nebula:0

Update initramfs:

# update-initramfs -u

and now .... :-))

cryptsetup luksOpen /dev/md0 md0_decrypted
Enter passphrase for /dev/md0:
/dev/mapper/md0_decrypted is active.
type: LUKS1
cipher: aes-cbc-essiv:sha256
keysize: 256 bits
device: /dev/md0
offset: 6144 sectors
size: 23435569152 sectors
mode: read/write
cryptsetup status md0_decrypted exit status 0
Scanning for volume groups
Reading all physical volumes. This may take a while...
Found volume group "ubuntu-vg" using metadata type lvm2
Found volume group "vg_storage" using metadata type lvm2
3 logical volume(s) in volume group "vg_storage" now active
VG Name vg_storage
LV Name lv_backups
LV Name lv_data
LV Name lv_media


I checked the filesystem and it's all right.

As I was saying for 12 years:
The strength of linux does not lie in the kernel as many say! But in the community that supports it!

Many thanks to all those who made this rescue possible.

And sorry for the long post.

Best regards

Adriano

I am amazed that you were able to recover from that rebuild. I would have expected issues with Q-parity chunks being mistaken for data and data being overwritten with Q-parity to have scrambled things thoroughly. But, I am very glad to head that you got it all back. I admire your persistence.