undo stupid pvcreate -ff

soliton · 06-20-2014, 03:08 PM

hello,
i did something incredibly stupid.

i have 2 disk arrays configured with lvm that i keep synced regularly for backup.
one of arrays had a broken hard drive thats why i took it down and wanted to replace it.

unfortunately due to wrongly connected usb drives and mostly personal stupidity i executed pvcreate -ff on one of the good drives
now i have 2 lvm arrays and cant not use one of them because one is broken and the other one was destroyed by me

is it possible to undo the pvcreate -ff?

the configuration is as follows:

4 physical devices
2x 2TB
2x 3TB
stripe config for the lvm
on top of that cryptsetup
the pvcreate -ff was executed on one of the 2tb drives.

i would be so greatful for help

smallpond · 06-20-2014, 03:34 PM

pvcreate overwrites a few sectors at the beginning of the partition or drive. You need to rewrite the pv label using pvcreate restoring the original uuid. If you have a backup, you can find it in /etc/lvm/backup/<lvm-name>.

soliton · 06-20-2014, 03:48 PM

i do not have backups anymore since my backup drive broke down and i assumed i was working on the backup.
or are you refering to some kind of lvm meta data backup. how do i find out if i have that?

thank u so much for replying.

if i restore the uuid is that enough? or do i have to write the connection to the volume group as well?
and how do i do either of those.

soliton · 06-20-2014, 03:58 PM

ok it would appear that i have a backup of the lvm group data at least on one server

since the group exists of 4 devices i am not certain how to proceed

pvcreate --restorefile /etc/lvb/backup/group4 --uuid <my missing uuid> <the device i executed pvcreate on?>

would that be correct?

smallpond · 06-20-2014, 04:21 PM

Sounds good. I've never had to do this, myself.

soliton · 06-20-2014, 04:25 PM

i copy the disk via dd right now which will take a while over usb2
i will let you know how it turned out
thx for help

rknichols · 06-20-2014, 07:50 PM

If your /etc/lvm/archive directory is intact, you have all the info you need, including the UUID for pvcreate and the file needed for vgcfgrestore to put everything back together.

soliton · 06-23-2014, 06:44 PM

it did not work out completely
i managed to get recreate the group with vgcfgrestore

but it seems that the data from the disk is missing / not being read

i have a stripe config:
stripes = [
"pv2", 0,
"pv3", 0,
"pv0", 0,
"pv1", 0
]

the stripe segments are 4mb and it looks like that every 16 mb there is a 4mb piece missing.
it just contains random data.
the drive is encrypted so i would expect plain 0 data to be displayed as random. but why?
what went wrong.

additional information: i copied the entire disk via dd. after that the disk broke down and is not readable any more ;(
but i used the backup for recreating the group.

rknichols · 06-23-2014, 09:12 PM

Please post the .vg file from /etc/lvm/archive that you used for the vgcfgrestore, and which should contain the description, "Created *before* executing pvcreate ...".

soliton · 06-23-2014, 11:24 PM

this is the original vg file

Quote:

# Generated by LVM2 version 2.02.84(2) (2011-02-09): Mon Apr 1 03:54:07 2013

contents = "Text Format Volume Group"
version = 1

description = "Created *after* executing '/sbin/vgs --noheadings -o vg_name'"

creation_host = "server" # Linux server 3.4.33-2.24-desktop #1 SMP PREEMPT Tue Feb 26 03:34:33 UTC 2013 (5f00a32) x86_64
creation_time = 1364781247 # Mon Apr 1 03:54:07 2013

group3 {
id = "GUcW9k-KQuZ-hgwV-12n0-DIoq-VRNc-HoHQt8"
seqno = 2
status = ["RESIZEABLE", "READ", "WRITE"]
flags = []
extent_size = 8192 # 4 Megabytes
max_lv = 0
max_pv = 0

allocation_policy = "anywhere"
metadata_copies = 0

physical_volumes {

pv0 {
id = "TWzROW-2SBH-wPGJ-fZsw-wvOO-FbKB-qWz9eO"
device = "/dev/sde" # Hint only

status = ["ALLOCATABLE"]
flags = []
dev_size = 3907024896 # 1.81935 Terabytes
pe_start = 2048
pe_count = 476931 # 1.81935 Terabytes
}

pv1 {
id = "Ghkze1-OKwj-Qhfa-CIQ1-nOvK-OY0i-XJLWEw"
device = "/dev/sdd" # Hint only

status = ["ALLOCATABLE"]
flags = []
dev_size = 3907024896 # 1.81935 Terabytes
pe_start = 2048
pe_count = 476931 # 1.81935 Terabytes
}

pv2 {
id = "DWMW3s-iBBe-ttvc-hZk6-yJZi-s9Zn-p1nrgx"
device = "/dev/sdc" # Hint only

status = ["ALLOCATABLE"]
flags = []
dev_size = 5860533168 # 2.72902 Terabytes
pe_start = 2048
pe_count = 715396 # 2.72902 Terabytes
}

pv3 {
id = "LLxAdX-WyfM-3Hx5-wOqR-Ia5L-mhv5-uKjNEr"
device = "/dev/sdb" # Hint only

status = ["ALLOCATABLE"]
flags = []
dev_size = 5860533168 # 2.72902 Terabytes
pe_start = 2048
pe_count = 715396 # 2.72902 Terabytes
}
}

logical_volumes {

store3 {
id = "P317P7-DJXS-y9cC-au00-46eD-QDpO-X7ODPb"
status = ["READ", "WRITE", "VISIBLE"]
flags = []
allocation_policy = "anywhere"
segment_count = 2

segment1 {
start_extent = 0
extent_count = 1907724 # 7.27739 Terabytes

type = "striped"
stripe_count = 4
stripe_size = 8192 # 4 Megabytes

stripes = [
"pv2", 0,
"pv3", 0,
"pv0", 0,
"pv1", 0
]
}
segment2 {
start_extent = 1907724
extent_count = 476928 # 1.81934 Terabytes

type = "striped"
stripe_count = 4
stripe_size = 8192 # 4 Megabytes

stripes = [
"pv2", 476931,
"pv3", 476931,
"pv2", 596163,
"pv3", 596163
]
}
}
}
}

soliton · 06-23-2014, 11:45 PM

i did a
pvcreate --restorefile /home/sol/group3 --uuid <uuid> /dev/sd<...>

on every disk
for the disk i was copying with dd, i used the copy. the copy is on a 3tb disk. the original is a 2tb
is that a problem maybe?

then i did a

vgcfgrestore -f /home/sol/group3 group3

i rebooted (just to be sure)

started the lv (cryptsetup + mount)

and there were all my files, except that the data in the files is corrupt now ;(

soliton · 06-24-2014, 06:17 AM

i just copied all the data back to a identical 2tb hd

reinitialized the group

still one quarter of my data is missing ;(

pls help. i know i brought this on myself but its so frustrating. my data is slipping through my fingers ....

rknichols · 06-24-2014, 09:14 AM

Quote:

Originally Posted by soliton

this is the original vg file

Code:

# Generated by LVM2 version 2.02.84(2) (2011-02-09): Mon Apr 1 03:54:07 2013

contents = "Text Format Volume Group"
version = 1

description = "Created *after* executing '/sbin/vgs --noheadings -o vg_name'"

Please use [CODE]...[/CODE] tags and not [QUOTE]...[/QUOTE] tags for files so that formatting is preserved and subsequent quoting is not suppressed.

That looks like a file from /etc/lvm/backup. Are you certain that nothing in the LVM configuration changed in the 14+ months since that file was written? Do you not have a more recent file in /etc/lvm/archive with a description line, "Created *before* executing pvcreate ...", or did you perhaps run that ill-fated "pvcreate -ff" command from some rescue disk (If so, which one?) and not have that file saved in your root filesystem?

rknichols · 06-24-2014, 10:53 AM

Quote:

Originally Posted by soliton

i did a
pvcreate --restorefile /home/sol/group3 --uuid <uuid> /dev/sd<...>

on every disk
for the disk i was copying with dd, i used the copy. the copy is on a 3tb disk. the original is a 2tb
is that a problem maybe?

Why you would do a pvcreate on every disk is beyond my comprehension. When you find yourself stuck in a hole, the first rule is, "Stop digging!" Now I don't even know whether the stripe that is failing to decrypt is the one with the original problem or one of the others.

No, the 3TB vs. 2TB disks should not matter, perhaps depending on how you did the dd. What was the exact dd command you used?

BTW, which stripe (pv[0-3]) is the failing one?

Anything that does not decrypt properly is going to look like random data. You would need to look at the non-decrypted /dev/group3/store3 with hexedit to see whether the failing stripe appears as all-zero or as random data, the latter possibly being encrypted data or just the random numbers you filled the volume with (Did you do that?) while setting up the encryption.

soliton · 06-24-2014, 11:30 AM

i am not at the server right now but i can give you the following information

i am absolutely certain the group information did not change.
i never ever changed a vg. i only created new groups and i allways give it a new number
i do have several group archives but as far as i can tell they only differ in which pv is flaged as missing and which not

i moved the physical usb disks to a different server to do the maintainance and i compared the backup / archive from the new server with the original one and they are identical. but i will post newer versions when i get back to the server just to be sure.

first i did only

Code:

pvcreate --restorefile

on the pv that has had overwritten meta data at first.
but then i got the error message that the group is out of sync.

so i did

Code:

pvcreate --restorefile

on all drives.
i was very careful.
first i executed vgdisplay
adding one physical drive at a time i allways checked the uuid and dev path /dev/sdb /dev/sdc ...
i am pretty certain that i did not do a mistake at that step. it may be a unneccessary step but im sure i did not screw that one up.

after

Code:

vgcfgrestore /home/sol/Downloads/group3 group3

group was online again.

the file /home/sol/Downloads/group3 was copied from the original server just to be sure

i did not take the group offline with

Code:

vgchange -an group3

was that a mistake?

the exact dd was:

Code:

dd /dev/sd(x) /dev/sd(y) -bs 64K

x and y standing for the drive letter. i am not sure which one they were at the time
but i allways took every other drive offline and brought drives online one at a time to check which drive letter they were

the next thing i would check is if it makes a difference if i recreate the pv on the original server.
it has a older version of opensuse running. the new server has mint 13.10
maybe it makes a difference.

i read something about a offset issue when doing

Code:

pvcreate --restorefile

that was fixed earlier this year.
maybe there is a problem with the offset?

i will try to find out what offset is being used for my drive right now.

thank you very much for helping me.
i really hope we can safe my data.