i found something

the 4N+2 parts all look like random data

however!

the other parts dont. they are just plain 0 until

this is the first non 4N + 2 part that looks like random data.

that means something doesnt it?

shows random data while the same command on the other drives shows some structure
sdc is currently the faulty drive

even if i had taken a wrong drive at some step the head of the drive should be the same as all drives i have here are pvs
i am beginning to suspect that there might be something wrong with the usb hub i used to copy the data.
it gets very hot and i had long period of 100% cpu use while copying the data.
right now i am trying to make more backups and i also get 100% cpu.
i dont get the cpu use when i plug the drive directly to the usb 3 plug

this is very odd.
together with the odd random data that came from nowhere this indicates that something happened to the data while copying it

eather that or the drive was so faulty while copying it that i read faulty data

whats odd thou is when i finished copying the drive via dd it was successfully recogniced as a pv.
also the 2nd copy to a 2tb drive was recogniced as a pv

i have no explaination where the random data on the 4M block 2 comes from.

When reading from the raw drive (/dev/sd[b-e]), the first 1MB of each drive ("pe_start = 2048" sectors) would not have been affected by the encryption. Whether that data appears random depends on what the drive previously had on it. At a minimum there must be a PV header beginning with the ASCII string "LABELONE" somewhere in the first 4 sectors (probably the second) on each drive. If you used LUKS encryption, the drive with the first stripe (pv2) would have a ~1MB LUKS header beginning at LBA 2048. Other than that, any sectors that have been written since encryption was set up would appear random. Sectors that have not been written would have whatever was previously on the drive.

When reading from /dev/group3/store3, the only non-random sectors would be those that were never written with encrypted data. It doesn't make sense to me that there would be 128MB of all-zeros (except for the bad drive) there. Just how was that encryption set up? I've been assuming LUKS encryption of the group3-store3 LV. Please let me know if it was something different.

This line is saying that the 3TB drive that you are using has 4K sectors. Did the old drive have 4K or 512-byte sectors?

uh, how do i find out?

its a pretty straigt forward luks encryption. i read several articles about it and choose the strongest encryption that was recommended. other than that just plain luksFormat

The drives would all have 512-byte logical sector size. Current kernels do not support 4KB logical sector size. Whether the physical sector size is 512 bytes or 4KB should not be relevant.

i just tried to restore the pv on the original server. i hoped that the problem may be a result from a incopatibility between different lvm versions.

but no ;(

my last hope is gone. so sad.

so sad ....

The only thing I can think of to try is to assume that the data currently on /dev/sdc is basically correct but is offset by some number of sectors. Here is what I would do to try to resolve that.

1. On an ext4 filesystem with at least 3GB of free space, create a sparse file 16MB larger than the total size of sdc.
   Code:

   truncate --size=$((3907024896*512 + 16*1024*1024)) file.img

2. Copy about 1GB from the beginning of sdc to the sparse file at offset 16MB.
   Code:

   dd if=/dev/sdc bs=1M count=1024 seek=16 conv=notrunc of=file.img

3. If you forgot to use the "conv=notrunc" option in step2, use truncate again to restore the size used in step 1.

Now you can set up a loop device for that file, beginning at offset 0 and successively increasing the offset by 512 bytes, each time running pvcreate on that loop device and then vgcfgrestore, and seeing of you can find an offset that eliminates the 4MB blocks of random data in /dev/group3/store3. You won't be able to mount the filesystem because you don't have the full 2TB available, but you should be able to look at some data within the 1GB that you do have for that stripe (not too near the beginning -- that might have been clobbered).

1. Offset=0
2. losetup -o $Offset /dev/loop0 file.img
3. pvcreate --uuid TWzROW-2SBH-wPGJ-fZsw-wvOO-FbKB-qWz9eO --restorefile /home/sol/group3 /dev/loop0
4. vgcfgrestore -f /home/sol/group3 group3

Then you would use cryptsetup to decrypt the LV and see if most of the blocks at sector offsets 4N+2 in the decrypted device stop looking random. If not, delete the loop device, add 512 to the offset, and try again.

I'm sure I've left out some necessary steps of activating/deactivating the volume group and removing the PV so that the loop device will not appear busy. This just isn't something I've got a reasonable way to test myself.