[SOLVED] LVM problem - Need to remove PV, impossible with pvremove
CentOSThis forum is for the discussion of CentOS Linux. Note: This forum does not have any official participation.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
LVM problem - Need to remove PV, impossible with pvremove
This problem surfaced while trying to solve another problem. Let me explain.
I have an existing LVM consisting of 4 disks. Suddenly 2 disks failed with I/O errors, making the superblock unreadable (filesystem = XFS). I purchased 2 new disks, intending to replace the failing disks by making copies via ddrescue. I wanted to add the 2 new disks to my VG, then run vgcfgbackup and vgcfgrestore in order to regain a working (though still half empty) VG. After copying the falings disks onto the new ones I should be back in business.
However, I goofed by adding the 2 new disks twice. Now I have a VG consisting of 6 PV's (2 original ones that are still OK sdf1 and sdg1, 2 new disks sdi1 and sdj1, and 2 fake disks). I tried to remove both fake disks but pvremove will not let me do it.
I am absolutely certain that both uuid's that are now reported as missing, are empty. How can I get rid of them, so that I can reconvene my restore?
Here's some output:
PHP Code:
[root@san2 ~]# pvscan PV /dev/sda3 VG clearos lvm2 [13,71 GiB / 0 free] WARNING: Device for PV H3cLlM-6Gnk-HK8W-ax7g-LAtT-0JpR-Fc7OMA not found or rejected by a filter. WARNING: Device for PV ZP56SO-Ozlj-r7vG-4R8O-JR8n-Yab6-MW7LBu not found or rejected by a filter. PV /dev/sde1 VG AV lvm2 [<3,64 TiB / 0 free] PV /dev/sdf1 VG AV lvm2 [<2,73 TiB / 0 free] PV /dev/sdi1 VG AV lvm2 [<3,64 TiB / 0 free] PV /dev/sdj1 VG AV lvm2 [<3,64 TiB / 0 free] PV [unknown] VG AV lvm2 [<4,55 TiB / <4,55 TiB free] PV [unknown] VG AV lvm2 [<4,55 TiB / <4,55 TiB free] Total: 8 [<23,44 TiB] / in use: 8 [<23,44 TiB] / in no VG: 0 [0
PHP Code:
[root@san2 ~]# vgcfgrestore -v --file temp13.txt AV Cannot restore Volume Group AV with 2 PVs marked as missing. Restore failed.
PHP Code:
[root@san2 ~]# pvremove uuid=H3cLlM-6Gnk-HK8W-ax7g-LAtT-0JpR-Fc7OMA WARNING: Device for PV H3cLlM-6Gnk-HK8W-ax7g-LAtT-0JpR-Fc7OMA not found or rejected by a filter. WARNING: Device for PV ZP56SO-Ozlj-r7vG-4R8O-JR8n-Yab6-MW7LBu not found or rejected by a filter. Device uuid=H3cLlM-6Gnk-HK8W-ax7g-LAtT-0JpR-Fc7OMA not found.
You don't need to call pvremove on a missing volume. pvremove wipes the PV label on an existing volume. You already wiped the first labels by doing pvcreate a second time. If you added the fake PVs to your volume group, then you need to reduce the volume group using vgreduce.
The warning for PVs not found is due to the cached copy of your metadata. Running "pvscan --cache" should refresh it.
You don't need to call pvremove on a missing volume. pvremove wipes the PV label on an existing volume. You already wiped the first labels by doing pvcreate a second time. If you added the fake PVs to your volume group, then you need to reduce the volume group using vgreduce.
The warning for PVs not found is due to the cached copy of your metadata. Running "pvscan --cache" should refresh it.
That made a lot of sense but it only worsened the situation:
PHP Code:
[root@san2 ~]# pvscan --cache WARNING: PV w7WVft-LUpq-sjEc-IJFm-c41t-Qv9W-PYzIlM on /dev/sdi1 was already found on /dev/sde1. WARNING: Disabling lvmetad cache which does not support duplicate PVs. WARNING: PV JAyBXE-aZrQ-dVUO-UB0W-PQuT-GTrT-IEYT2w on /dev/sdj1 was already found on /dev/sdf1. WARNING: Not using lvmetad because duplicate PVs were found. [root@san2 ~]# pvscan WARNING: PV w7WVft-LUpq-sjEc-IJFm-c41t-Qv9W-PYzIlM on /dev/sdi1 was already found on /dev/sde1. WARNING: Disabling lvmetad cache which does not support duplicate PVs. WARNING: PV JAyBXE-aZrQ-dVUO-UB0W-PQuT-GTrT-IEYT2w on /dev/sdj1 was already found on /dev/sdf1. WARNING: Not using lvmetad because duplicate PVs were found. WARNING: PV w7WVft-LUpq-sjEc-IJFm-c41t-Qv9W-PYzIlM on /dev/sdi1 was already found on /dev/sde1. WARNING: PV JAyBXE-aZrQ-dVUO-UB0W-PQuT-GTrT-IEYT2w on /dev/sdj1 was already found on /dev/sdf1. WARNING: PV w7WVft-LUpq-sjEc-IJFm-c41t-Qv9W-PYzIlM prefers device /dev/sde1 because device size is correct. WARNING: PV JAyBXE-aZrQ-dVUO-UB0W-PQuT-GTrT-IEYT2w prefers device /dev/sdf1 because device size is correct. WARNING: PV w7WVft-LUpq-sjEc-IJFm-c41t-Qv9W-PYzIlM on /dev/sdi1 was already found on /dev/sde1. WARNING: PV JAyBXE-aZrQ-dVUO-UB0W-PQuT-GTrT-IEYT2w on /dev/sdj1 was already found on /dev/sdf1. WARNING: PV w7WVft-LUpq-sjEc-IJFm-c41t-Qv9W-PYzIlM prefers device /dev/sde1 because of previous preference. WARNING: PV JAyBXE-aZrQ-dVUO-UB0W-PQuT-GTrT-IEYT2w prefers device /dev/sdf1 because of previous preference. Couldn't find device with uuid 8C8y2a-GXmN-tPNx-Jha2-Rea1-cqav-AfyfNm. Couldn't find device with uuid JTdK18-uuaq-8U4t-RQ2I-etKd-jUF5-03xIjk. Couldn't find device with uuid H3cLlM-6Gnk-HK8W-ax7g-LAtT-0JpR-Fc7OMA. Couldn't find device with uuid ZP56SO-Ozlj-r7vG-4R8O-JR8n-Yab6-MW7LBu. WARNING: Couldn't find all devices for LV AV/AV while checking used and assumed devices. PV /dev/sde1 VG AV lvm2 [<3,64 TiB / 0 free] PV /dev/sdf1 VG AV lvm2 [<2,73 TiB / 0 free] PV [unknown] VG AV lvm2 [<3,64 TiB / 0 free] PV [unknown] VG AV lvm2 [<3,64 TiB / 0 free] PV [unknown] VG AV lvm2 [<4,55 TiB / <4,55 TiB free] PV [unknown] VG AV lvm2 [<4,55 TiB / <4,55 TiB free] PV /dev/sda3 VG clearos lvm2 [13,71 GiB / 0 free] Total: 7 [<23,44 TiB] / in use: 7 [<23,44 TiB] / in no VG: 0 [0 ]
The 2 new disks /dev/sdi and /dev/sdj were correctly registered before the cache was cleared. They correctly mirrorred the UUID's of both disks that had gone bad. After clearing the cache these new disks suddenly got wrong UUID's, i.e. of /dev/sde1 and /dev/sdf1, which are the other 2 original disks, the disks having no problems. Can I at least go back to the situation before the cache clearance? As the original post demonstrates, I then had 4 correctly registered PVs (plus 2 fakes).
Note: vgreduce will not work untill the metadata are corrected.
But I can't restore this file. The 2 new devices /dev/sdi and /dev/sdj seem to have lost their LVM info. I tried to pvcreate these devices again but that failed. How do I correctly restore the backup file?
First I think you need to figure out what caused the duplicate uuid message. Do you have multiple paths to the drives? Did you not remove the failed drives after ddrescue? Or is it possible the wrong drives were copied?
I understand, but I would still face the same problem as the metadata would still be mixed up. LVM doesn't let me use any LVM command before this is rectified, and it won't even let me rectify. I really need to do something more definitive with the metadata.
I do have the correct number of devices connected as lsblk shows, and I do have a correct lvm config file that worked in the beginning. Isn't there a way to get rid of all caches etc (LVM won't let me run pvscan --cache) and just use the bare config file that used to work before LVM mixed it all up? Thanks for trying to help me out.
In case you did not know, those files in /etc/lvm/archive and /etc/lvm/backup are exactly that, archive and backup. They are not controlling. The entire state of the volume group is recorded on each PV in the volume group in its LVM header. The file in /etc/lvm/backup is just a formatted and commented version of the ASCII text that begins in the 9th 512-byte sector of that header.
It's not clear just how you got into your current state, but you can unconditionally clear all your LVM headers by running "dd if=/dev/zero bs=1k count=10 of=/dev/{xxxx}" on each of your physical volumes. You will then need to restore those PVs by running pvcreate with the "--uuid" and "--restorefile" options using your known good configuration file as the data source. That should get you into a state where you can do a successful vgcfgrestore.
In case you did not know, those files in /etc/lvm/archive and /etc/lvm/backup are exactly that, archive and backup. They are not controlling. The entire state of the volume group is recorded on each PV in the volume group in its LVM header. The file in /etc/lvm/backup is just a formatted and commented version of the ASCII text that begins in the 9th 512-byte sector of that header.
It's not clear just how you got into your current state, but you can unconditionally clear all your LVM headers by running "dd if=/dev/zero bs=1k count=10 of=/dev/{xxxx}" on each of your physical volumes. You will then need to restore those PVs by running pvcreate with the "--uuid" and "--restorefile" options using your known good configuration file as the data source. That should get you into a state where you can do a successful vgcfgrestore.
Many thanks for the explanation. This clarified my situation and led me to successfully solve the problem.
I was hoping that the "wipe" option of (newer) fdisk would erase the LVM meta-data, but a quick test proved disappointing, so we'll have to keep using dd for the foreseeable future.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.