Disaster with LVM

h725 · 04-26-2008, 05:30 AM

Hi,

I had an LVM with two PV, md1 and md2 (both RAID1).
The VG was called "archive", and the LVM was "MaxtorLVM".
File system was ext3.
Cause of my inexperience with LVM, I deleted all the stuff (with lvremove, pvremove and vgremove command) but without overwriting the content of md devices or reuse them.
I thought that he was possible rebuild all the content of LVM, by recreate PV, VG and LV with the same size - parameter - uuid and so on..

Unfortunately, I get an error when I try to mount the LVM device:

# mount /dev/mapper/archive-MaxtorLVM /lvm/
mount: you must specify the filesystem type

# mount -t ext3 /dev/mapper/archive-MaxtorLVM /lvm/
mount: wrong fs type, bad option, bad superblock on /dev/mapper/archive-MaxtorLVM,
missing codepage or other error
In some cases useful info is found in syslog - try
dmesg | tail or so

dmesg says:

Apr 26 12:13:22 backup kernel: VFS: Can't find a valid ext3 filesystem on dev dm-0.

cat /proc/mdstat is ok.
Other info:
Debian stable, kernel 2.6.24.4

Having the backup in /etc/lvm/archive and /etc/lvm/backup, It's possibile to recover the content of the old LVM?

Thanks

rayfordj · 04-27-2008, 05:11 PM

If only the LVM metadata has been modified and the actual filesystem data has been left in-tact (meaning no process has caused any modification) and you have an available config from archive or backup then you might be able to recover the LV by using vgcfgrestore. Consult the man page for its use.

I've used this to "repair" corrupt on-disk LVM metadata but never tried it after lv-, vg-, pvremove.

Hope this helps.

Edit 1
Ok, so you got me curious.
A quick run-down of what I did and my results.

presented a ~20GB iscsi target to my system (sdd)
created single LinuxLVM primary partition sdd1

Code:

pvcreate /dev/sdd1
vgcreate testVG /dev/sdd1
lvcreate -l 100%VG -n testlV /dev/testVG
mke2fs -j /dev/testVG/testLV
mount /dev/testVG/testLV /mnt/test
cp /tmp/lvm.tgz /mnt/test/
umount /mnt/test
lvremove /dev/testVG/testLV
vgremove /dev/testVG
pvremove /dev/sdd1
pvcreate /dev/sdd1
vgcreate /dev/test2VG
lvcreate -l 100%VG -n test2LV /dev/test2VG
mount /dev/test2VG/test2LV /mnt/test

get the same fs error you are seeing

Code:

lvremove /dev/test2VG/test2LV
vgremove /dev/test2VG
pvremove /dev/sdd1

back to a partition (sdd1) with no LVM metadata on it
at this point i reviewed the testVG_00*.vg in /etc/lvm/archive to find the one that has the correct information that i want/need.
mine happened to be "testVG_00002.vg", yours may (and will likely) be different.

grab the pv uuid from there for the next command

Code:

pvcreate --restorefile /etc/lvm/archive/testVG_00002.vg --uuid N9iohj-6Rhd-Tv6S-8Wo9-NtnZ-dpj3-JVcF0G  /dev/sdd1

proceeding, restore the vg metadata, confirm the vg info, activate the lv, and attempt mount

Code:

vgcfgrestore  -f  /etc/lvm/archive/testVG_00002.vg -v testVG
vgdisplay -v /dev/testVG
vgchange -ay /dev/testVG
mount /dev/testVG/testLV /mnt/test

mount fails again with your error

run an fsck on it (i used ext3 for my fs)

Code:

e2fsck /dev/testVG/testLV

you will in all likely-hood encounter "problems" with the filesystem, i said yes to all questions to fix or correct a problem;
ran an fsck a 2nd time, came back clean

Code:

#mount /dev/testVG/testLV /mnt/test
#sha1sum /mnt/test/lvm.tgz /tmp/lvm.tgz
  a73297ecb368a1b0aeea98b4f39c37363fd7f49f  /mnt/test/lvm.tgz
  a73297ecb368a1b0aeea98b4f39c37363fd7f49f  /tmp/lvm.tgz

the file on the testVG that i just restored matches the original when output compared of sha1sum of the two files.

I can only hope you are as successful.

Hope this helps even more.

h725 · 05-01-2008, 02:46 AM

Hi,

sorry I'm late, your answer was very useful for me

I did the same procedure and all data have been recovered!

Thanks a lot!!

rayfordj · 05-01-2008, 07:11 AM

Great news!
I'm glad it worked out for you!

travisn000 · 04-18-2010, 01:09 PM

@rayfordj

Thank you for your wonderful post! With your example I was able to successfully recover ~ 650 MB of family photos, music, etc.

Truly a life saver!
(..now I keep a compressed archive of my /etc/lvm directory in the cloud for easy access

)

Thanks again.

Travis

PS.
..for any others that may be in need, there is also a similar solution involving multiple physical volumes here:

http://codeworks.gnomedia.com/archiv.../lvm_recovery/

hasretg · 06-28-2011, 03:57 AM

jtholb · 11-26-2011, 08:16 AM

I don't suppose by some miracle you guys are still following this thread...and I'm sorry for bumping an old thread...

I feel like I'm so close to being able to get my data back, but I'm stuck. Everything goes smoothly

Code:

vgcfgrestore  -f  /etc/lvm/archive/testVG_00002.vg -v testVG
vgdisplay -v /dev/testVG
vgchange -ay /dev/testVG

but with

Code:

mount -t ext4 /dev/group0/home /home
mount: wrong fs type, bad option, bad superblock on /dev/mapper/group0-home,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

It's the just the modified version of

Code:

mount /dev/testVG/testLV /mnt/test

It's definitely ext4 and I've tried to restore the superblock with

Code:

e2fsck -b

but without any luck and now I'm at the point that I'm not sure what to do. Any help would be so massively appreciated.

makyo · 11-26-2011, 09:33 AM

Hi, jtholb.

Welcome to LQ.

Yes, some people are subscribed, but this is a very old thread.

In general, you are far more likely to receive more views of your question (and so more answers and discussion) if it is in a new thread. It's rarely the case that 2 problems are exactly the same, so a fresh thread is almost always your best bet. If you think the problems are close, you could refer to this thread with a link.

Best wishes ... cheers, makyo

_bsd · 12-03-2011, 06:45 AM

At this point I would suggest buying one of the commercial data recovery apps.

I had a similar error with my volume group, with a successful vgcfgrestore, except I was using xfs, not ext3/4

Writing the VG metadata apparently touches a block or two shared by filesystem metadata.

I used one of the "deep scan" data recovery apps (wasn't expensive either, less than $100) and the deep scan found all of my data (missing some/most filenames).

jtholb · 12-03-2011, 07:13 AM

Sorry for bumping an old thread. In the end I figured it out and all my data was still intact--lucky me. For a more thorough explanation of it all you can go to http://ubuntuforums.org/showthread.php?t=1886343 (I didn't actually get any replies, so it ended up being entirely my own posts of what was happening). In the end it was basically three commands that fixed things for me.

Code:

mdadm --create --metadata=0.9 --raid-devices=3 --chunk=64 --level=5 --assume-clean /dev/md127 /dev/sdb1 /dev/sdc1 missing --uuid=d6bfa8db:624bcebf:cd0fe2e0:41039747

pvcreate -u sftW7n-JaLc-l85n-IcJg-3fqW-Dd9k-NxyKjK /dev/md127

vgcfgrestore -f /etc/lvm/archive/group0_00001.vg group0

Thanks you guys.