cannot get overflown LVM thinpool to activate

crackers8199 · 08-20-2019, 05:43 PM

i have 3x 10tb disks in my server, and one of the three was dedicated solely to my plex media (for now). i had a lvm thinpool and stupidly created a drive in the thinpool for 10tb (which was too big, obviously). now i've filled the drive and am getting io errors, and can't reactivate it. i also can't repair it because there's no space left on the disk.

how can i fix this and get the data off the disk if i need to format and start over? i have enough room elsewhere to copy files to if i can get them mounted, but i've had no luck figuring out how to get the drive activated and mounted to get the data off of it and move it elsewhere...

i've added a 256GB usb stick and an additional 8tb drive that are completely empty in an attempt to extend this thing and get it mounted so that i can get the data off of it, wipe it and start over fresh, but i'm having absolutely no luck and i'm just getting extremely frustrated at this point. i'm going on about 3 weeks of trying to fix this thing and i have gotten absolutely nowhere.

last point before someone hopefully can help me: i do have what i believe is a valid metadata dump...but i'm getting nowhere trying to get it to work. i've gotten some combination of device or resource busy, or invalid argument errors...at this point i honestly don't even know which goes with which. i've tried everything i can think of.

Firerat · 09-03-2019, 07:21 AM

I assume you have tried

Code:

vgchange -ay Vol_Group

you may need to use --activationmode partial|degraded
but they are usually for when a pv is missing

it would probably help if you listed the commands you have tried

Code:

history | grep -E "(vg|lv|pv)[a-Z]"

crackers8199 · 09-03-2019, 12:55 PM

Quote:

Originally Posted by Firerat

I assume you have tried

Code:

vgchange -ay Vol_Group

you may need to use --activationmode partial|degraded
but they are usually for when a pv is missing

it would probably help if you listed the commands you have tried

Code:

history | grep -E "(vg|lv|pv)[a-Z]"

vgchange and lvchange are the two commands i've been trying to activate it. i always end up with one error or another.

here's another forum post i wrote up for this on another forum:

here's where i'm at. initially, after trying to run a repair, i got an error similar to the following:

device-mapper: resume ioctl on (252:3) failed: Invalid argument
Unable to resume vg02-pool0-tpool (252:3)
device-mapper: resume ioctl on (252:3) failed: Invalid argument
Unable to resume vg02-pool0-tpool (252:3)
along with this there were the following lines in dmesg:

[191450.356266] device-mapper: thin: Data device (dm-8) discard unsupported: Disabling discard passdown.
[191450.356270] device-mapper: thin: 253:9: metadata device (4145152 blocks) too small: expected 4161600
[191450.356275] device-mapper: table: 253:9: thin-pool: preresume failed, error = -22
that led me to the following link:

https://bugs.launchpad.net/ubuntu/+s...2/+bug/1625201

the most recent thing i've tried to do is attempt to restore my metadata using the procedures outlined there.

thin_dump /dev/mapper/vg02-pool0_tmeta > lvm_meta_dum
lvcreate -n pool0meta2 -L 12G vg02
thin_restore -i lvm_meta_dump -o /dev/mapper/vg02-pool0meta2
lvconvert --thinpool vg02/pool0 --poolmetadata vg02/pool0meta2
this has left me the following error when trying to extend or activate:

root@proxmox:~# lvextend /dev/pve2/data2 /dev/sde1
device-mapper: create ioctl on pve2-data2_tmetaLVM-Zg3oPyGdhFhGZJ7ixqUTMLj61ITjaVbGHweBLt5PoD9nl0Z92YcC63HLr9SvHLNf-tmeta failed: Device or resource busy
Failed to activate pve2/data2.
or

root@proxmox:~# lvchange -ay pve2
device-mapper: create ioctl on pve2-data2_tmetaLVM-Zg3oPyGdhFhGZJ7ixqUTMLj61ITjaVbGHweBLt5PoD9nl0Z92YcC63HLr9SvHLNf-tmeta failed: Device or resource busy
device-mapper: create ioctl on pve2-data2_tmetaLVM-Zg3oPyGdhFhGZJ7ixqUTMLj61ITjaVbGHweBLt5PoD9nl0Z92YcC63HLr9SvHLNf-tmeta failed: Device or resource busy

rknichols · 09-03-2019, 01:26 PM

If worst comes to worst it should be possible to use the metadata info to create a dmsetup mapping that constructs a device that holds the chunks of your LV, thus allowing you to extract your data. Because of the way chunks get allocated in a thinpool it's going to be somewhat a mess, and really messy if stripes, snapshots, or mirrors are involved, but should still be possible.

I'd need to see that metadata in order to help with that. It's probably too large to post inline here, and I suspect you need a higher post count in order to create an attachment, so you may need to put it in a pastebin somewhere and post a link.

Firerat · 09-03-2019, 01:53 PM

I'm getting confused now
I am rusty with lvm

Quote:

Code:

lvextend /dev/pve2/data2 /dev/sde1

shouldn't /dev/sde1 be made a PV added to the VG ( pve2 )
and then the LV can be extended

Code:

lvextend -L+10G /dev/pve2/data2
# to extend it 10G

oh ok
looking at lvextend's manpage the examples show adding /dev/sdk3
What is not clear is if pvcreate was used on sdk3 prior.

crackers8199 · 09-03-2019, 04:08 PM

Quote:

Originally Posted by Firerat

I'm getting confused now
I am rusty with lvm

shouldn't /dev/sde1 be made a PV added to the VG ( pve2 )
and then the LV can be extended

Code:

lvextend -L+10G /dev/pve2/data2
# to extend it 10G

oh ok
looking at lvextend's manpage the examples show adding /dev/sdk3
What is not clear is if pvcreate was used on sdk3 prior.

it was. i added the new drives via pvcreate.

root@proxmox:~# pvs
PV VG Fmt Attr PSize PFree
/dev/sda3 pve lvm2 a-- 9.10t 0
/dev/sdb1 pve2 lvm2 a-- 9.10t 15.99g
/dev/sdc1 pve-sdb lvm2 a-- 9.10t 0
/dev/sdd1 pve2 lvm2 a-- 233.25g 201.25g
/dev/sde1 pve2 lvm2 a-- 7.28t 7.28t

sdd is the 256gb usb stick, and sde is the new 8tb external. i'm actually going to return the 8tb and pick up a 10tb instead, as it's on sale now at best buy...but same difference.

crackers8199 · 09-03-2019, 04:24 PM

Quote:

Originally Posted by rknichols

If worst comes to worst it should be possible to use the metadata info to create a dmsetup mapping that constructs a device that holds the chunks of your LV, thus allowing you to extract your data. Because of the way chunks get allocated in a thinpool it's going to be somewhat a mess, and really messy if stripes, snapshots, or mirrors are involved, but should still be possible.

I'd need to see that metadata in order to help with that. It's probably too large to post inline here, and I suspect you need a higher post count in order to create an attachment, so you may need to put it in a pastebin somewhere and post a link.

it was too big to upload here or pastebin, so i put it on my dropbox and created a share link. here you go:

https://www.dropbox.com/s/0nxvid2bmq...90903.txt?dl=0

rknichols · 09-03-2019, 10:42 PM

OK, I've downloaded it. One more thing I'll need is the LVM metadata file for the VG in /etc/lvm/backup so that I have the correct parameters and names for the LVs. I may be too busy to do much on this for a day or two.

crackers8199 · 09-04-2019, 12:31 AM

Quote:

Originally Posted by rknichols

OK, I've downloaded it. One more thing I'll need is the LVM metadata file for the VG in /etc/lvm/backup so that I have the correct parameters and names for the LVs. I may be too busy to do much on this for a day or two.

that's totally fine, i really appreciate you taking the time to look at it. i've been trying for over a month and gotten absolutely nowhere, so for you to do this for me i'm extremely grateful.

here's the file from /etc/lvm/backup...

https://www.dropbox.com/s/ymig2a6qlpikha3/pve2?dl=0

rknichols · 09-05-2019, 07:51 PM

OK, let's try this. First, put this code in a file remapper.awk

Code:

BEGIN {
    devname="/dev/mapper/pve2-data2_tdata"
    o_end = -1
 }
func gmap(orig, len, target) {
    if(o_end >= 0 && orig > o_end) {
	print o_end*dbs, (orig-o_end)*dbs, "zero"
	o_end = orig
    }
    print orig*dbs, len*dbs, "linear", devname, target*dbs
    o_end = orig+len
}
match($0, "data_block_size=\"([1-9][0-9]*)\"", mm) {
    dbs = mm[1] + 0
    next
}
match($0, "single_mapping origin_block=\"(0|[1-9][0-9]*)\" data_block=\"(0|[1-9][0-9]*)\"", mm) {
    gmap(mm[1], 1, mm[2])
    next
}
match($0, "range_mapping origin_begin=\"(0|[1-9][0-9]*)\" data_begin=\"(0|[1-9][0-9]*)\" length=\"([1-9][0-9]*)\"", mm) {
    gmap(mm[1], mm[3], mm[2])
}

Then run

Code:

awk -f remapper.awk meta.bak.20190903.txt | dmsetup --readonly create dummy

With luck, and perhaps the application of a few unicorn tears, /dev/mapper/dummy should contain your old filesystem. Note the "--readonly" option recommended for safety. If it gets to the point that you think running fsck is necessary and would not cause further damage, then remove the "dummy" device and re-create it without that option. On second thought, that would be a terrible idea since writes to any addresses mapped to the "zero" target would be silently discarded. Think of the result should fsck try to fix anything and some of its writes were ignored. Leave it read-only. There are ways to set up a writable snapshot should it become necessary to do repairs.

(And yes, I know that script is not a robust way to parse XML. It works just fine for the file in question.)

syg00 · 09-05-2019, 08:29 PM

Well that appears pretty straight-forward ....

Nice - you continue to amaze me with the capabilities of device manager; which I thought was fading into history ....

rknichols · 09-05-2019, 09:42 PM

Quote:

Originally Posted by syg00

Nice - you continue to amaze me with the capabilities of device manager; which I thought was fading into history ....

"Fading into history"??? It's the direct interface to the device mapper that handles all the mappings for partitions, LVM, RAID, virtual disks..., basically anything that needs to map a block offset in a logical device into an offset in the next outer layer of the onion.

Yes the script is pretty straightforward. It took longer to set up a test environment and figure out the necessary incantation to get thin_dump to produce a map than it did to write and debug the script.

syg00 · 09-05-2019, 10:31 PM

My bad - I was thinking of "direct" interactions like dmraid. In some senses dmsetup seems a hang-over from times past.
These days almost everyone uses the higher level interfaces - hence threads like this pop up.

crackers8199 · 09-06-2019, 01:03 PM

Quote:

Originally Posted by rknichols

OK, let's try this. First, put this code in a file remapper.awk

Code:

BEGIN {
    devname="/dev/mapper/pve2-data2_tdata"
    o_end = -1
 }
func gmap(orig, len, target) {
    if(o_end >= 0 && orig > o_end) {
	print o_end*dbs, (orig-o_end)*dbs, "zero"
	o_end = orig
    }
    print orig*dbs, len*dbs, "linear", devname, target*dbs
    o_end = orig+len
}
match($0, "data_block_size=\"([1-9][0-9]*)\"", mm) {
    dbs = mm[1] + 0
    next
}
match($0, "single_mapping origin_block=\"(0|[1-9][0-9]*)\" data_block=\"(0|[1-9][0-9]*)\"", mm) {
    gmap(mm[1], 1, mm[2])
    next
}
match($0, "range_mapping origin_begin=\"(0|[1-9][0-9]*)\" data_begin=\"(0|[1-9][0-9]*)\" length=\"([1-9][0-9]*)\"", mm) {
    gmap(mm[1], mm[3], mm[2])
}

Then run

Code:

awk -f remapper.awk meta.bak.20190903.txt | dmsetup --readonly create dummy

With luck, and perhaps the application of a few unicorn tears, /dev/mapper/dummy should contain your old filesystem. Note the "--readonly" option recommended for safety. If it gets to the point that you think running fsck is necessary and would not cause further damage, then remove the "dummy" device and re-create it without that option. On second thought, that would be a terrible idea since writes to any addresses mapped to the "zero" target would be silently discarded. Think of the result should fsck try to fix anything and some of its writes were ignored. Leave it read-only. There are ways to set up a writable snapshot should it become necessary to do repairs.

(And yes, I know that script is not a robust way to parse XML. It works just fine for the file in question.)

i ended up with syntax errors...

Code:

awk: remapper.awk: line 13: syntax error at or near ,
awk: remapper.awk: line 17: syntax error at or near ,
awk: remapper.awk: line 21: syntax error at or near ,

it seemed like it was hung after that, so i killed it with ctrl-c. on running again, i got the syntax errors one more time but now also get:

Code:

device-mapper: create ioctl on dummy failed: Device or resource busy
Command failed

rknichols · 09-06-2019, 07:20 PM

I think that when you copied that script to a file you did it in a manner that the something in lines 13, 17, and 21 got interpreted by the shell and were not copied literally. Look at the awk script in an editor, and make sure it is exactly what I posted. The lines in question are the three "match(...)" calls.
See the subsequent posts.

I think you'll find that "dummy" now exists as an empty device. Use "dmsetup remove dummy" to get rid of it.