LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > CentOS
User Name
Password
CentOS This forum is for the discussion of CentOS Linux. Note: This forum does not have any official participation.

Notices


Reply
  Search this Thread
Old 08-18-2018, 11:43 AM   #1
roelvdh
LQ Newbie
 
Registered: Aug 2018
Location: Netherlands
Distribution: ClearOS
Posts: 26

Rep: Reputation: Disabled
LVM problem - Need to remove PV, impossible with pvremove


This problem surfaced while trying to solve another problem. Let me explain.
I have an existing LVM consisting of 4 disks. Suddenly 2 disks failed with I/O errors, making the superblock unreadable (filesystem = XFS). I purchased 2 new disks, intending to replace the failing disks by making copies via ddrescue. I wanted to add the 2 new disks to my VG, then run vgcfgbackup and vgcfgrestore in order to regain a working (though still half empty) VG. After copying the falings disks onto the new ones I should be back in business.
However, I goofed by adding the 2 new disks twice. Now I have a VG consisting of 6 PV's (2 original ones that are still OK sdf1 and sdg1, 2 new disks sdi1 and sdj1, and 2 fake disks). I tried to remove both fake disks but pvremove will not let me do it.
I am absolutely certain that both uuid's that are now reported as missing, are empty. How can I get rid of them, so that I can reconvene my restore?
Here's some output:

PHP Code:
[root@san2 ~]# pvscan
  
PV /dev/sda3   VG clearos         lvm2 [13,71 GiB 0    free]
  
WARNINGDevice for PV H3cLlM-6Gnk-HK8W-ax7g-LAtT-0JpR-Fc7OMA not found or rejected by a filter.
  
WARNINGDevice for PV ZP56SO-Ozlj-r7vG-4R8O-JR8n-Yab6-MW7LBu not found or rejected by a filter.
  
PV /dev/sde1   VG AV              lvm2 [<3,64 TiB 0    free]
  
PV /dev/sdf1   VG AV              lvm2 [<2,73 TiB 0    free]
  
PV /dev/sdi1   VG AV              lvm2 [<3,64 TiB 0    free]
  
PV /dev/sdj1   VG AV              lvm2 [<3,64 TiB 0    free]
  
PV [unknown]   VG AV              lvm2 [<4,55 TiB / <4,55 TiB free]
  
PV [unknown]   VG AV              lvm2 [<4,55 TiB / <4,55 TiB free]
  
Total[<23,44 TiB] / in use: [<23,44 TiB] / in no VG[
PHP Code:
[root@san2 ~]# vgcfgrestore -v --file temp13.txt AV
  
Cannot restore Volume Group AV with 2 PVs marked as missing.
  
Restore failed
PHP Code:
[root@san2 ~]# pvremove uuid=H3cLlM-6Gnk-HK8W-ax7g-LAtT-0JpR-Fc7OMA
  
WARNINGDevice for PV H3cLlM-6Gnk-HK8W-ax7g-LAtT-0JpR-Fc7OMA not found or rejected by a filter.
  
WARNINGDevice for PV ZP56SO-Ozlj-r7vG-4R8O-JR8n-Yab6-MW7LBu not found or rejected by a filter.
  
Device uuid=H3cLlM-6Gnk-HK8W-ax7g-LAtT-0JpR-Fc7OMA not found
 
Old 08-20-2018, 02:55 PM   #2
smallpond
Senior Member
 
Registered: Feb 2011
Location: Massachusetts, USA
Distribution: Fedora
Posts: 4,140

Rep: Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263
You don't need to call pvremove on a missing volume. pvremove wipes the PV label on an existing volume. You already wiped the first labels by doing pvcreate a second time. If you added the fake PVs to your volume group, then you need to reduce the volume group using vgreduce.

The warning for PVs not found is due to the cached copy of your metadata. Running "pvscan --cache" should refresh it.
 
Old 08-21-2018, 03:26 AM   #3
roelvdh
LQ Newbie
 
Registered: Aug 2018
Location: Netherlands
Distribution: ClearOS
Posts: 26

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by smallpond View Post
You don't need to call pvremove on a missing volume. pvremove wipes the PV label on an existing volume. You already wiped the first labels by doing pvcreate a second time. If you added the fake PVs to your volume group, then you need to reduce the volume group using vgreduce.

The warning for PVs not found is due to the cached copy of your metadata. Running "pvscan --cache" should refresh it.
That made a lot of sense but it only worsened the situation:

PHP Code:
[root@san2 ~]# pvscan --cache
  
WARNINGPV w7WVft-LUpq-sjEc-IJFm-c41t-Qv9W-PYzIlM on /dev/sdi1 was already found on /dev/sde1.
  
WARNINGDisabling lvmetad cache which does not support duplicate PVs.
  
WARNINGPV JAyBXE-aZrQ-dVUO-UB0W-PQuT-GTrT-IEYT2w on /dev/sdj1 was already found on /dev/sdf1.
  
WARNINGNot using lvmetad because duplicate PVs were found.
[
root@san2 ~]# pvscan
  
WARNINGPV w7WVft-LUpq-sjEc-IJFm-c41t-Qv9W-PYzIlM on /dev/sdi1 was already found on /dev/sde1.
  
WARNINGDisabling lvmetad cache which does not support duplicate PVs.
  
WARNINGPV JAyBXE-aZrQ-dVUO-UB0W-PQuT-GTrT-IEYT2w on /dev/sdj1 was already found on /dev/sdf1.
  
WARNINGNot using lvmetad because duplicate PVs were found.
  
WARNINGPV w7WVft-LUpq-sjEc-IJFm-c41t-Qv9W-PYzIlM on /dev/sdi1 was already found on /dev/sde1.
  
WARNINGPV JAyBXE-aZrQ-dVUO-UB0W-PQuT-GTrT-IEYT2w on /dev/sdj1 was already found on /dev/sdf1.
  
WARNINGPV w7WVft-LUpq-sjEc-IJFm-c41t-Qv9W-PYzIlM prefers device /dev/sde1 because device size is correct.
  
WARNINGPV JAyBXE-aZrQ-dVUO-UB0W-PQuT-GTrT-IEYT2w prefers device /dev/sdf1 because device size is correct.
  
WARNINGPV w7WVft-LUpq-sjEc-IJFm-c41t-Qv9W-PYzIlM on /dev/sdi1 was already found on /dev/sde1.
  
WARNINGPV JAyBXE-aZrQ-dVUO-UB0W-PQuT-GTrT-IEYT2w on /dev/sdj1 was already found on /dev/sdf1.
  
WARNINGPV w7WVft-LUpq-sjEc-IJFm-c41t-Qv9W-PYzIlM prefers device /dev/sde1 because of previous preference.
  
WARNINGPV JAyBXE-aZrQ-dVUO-UB0W-PQuT-GTrT-IEYT2w prefers device /dev/sdf1 because of previous preference.
  
Couldn't find device with uuid 8C8y2a-GXmN-tPNx-Jha2-Rea1-cqav-AfyfNm.
  Couldn'
t find device with uuid JTdK18-uuaq-8U4t-RQ2I-etKd-jUF5-03xIjk.
  
Couldn't find device with uuid H3cLlM-6Gnk-HK8W-ax7g-LAtT-0JpR-Fc7OMA.
  Couldn'
t find device with uuid ZP56SO-Ozlj-r7vG-4R8O-JR8n-Yab6-MW7LBu.
  
WARNINGCouldn't find all devices for LV AV/AV while checking used and assumed devices.
  PV /dev/sde1   VG AV              lvm2 [<3,64 TiB / 0    free]
  PV /dev/sdf1   VG AV              lvm2 [<2,73 TiB / 0    free]
  PV [unknown]   VG AV              lvm2 [<3,64 TiB / 0    free]
  PV [unknown]   VG AV              lvm2 [<3,64 TiB / 0    free]
  PV [unknown]   VG AV              lvm2 [<4,55 TiB / <4,55 TiB free]
  PV [unknown]   VG AV              lvm2 [<4,55 TiB / <4,55 TiB free]
  PV /dev/sda3   VG clearos         lvm2 [13,71 GiB / 0    free]
  Total: 7 [<23,44 TiB] / in use: 7 [<23,44 TiB] / in no VG: 0 [0   ] 
The 2 new disks /dev/sdi and /dev/sdj were correctly registered before the cache was cleared. They correctly mirrorred the UUID's of both disks that had gone bad. After clearing the cache these new disks suddenly got wrong UUID's, i.e. of /dev/sde1 and /dev/sdf1, which are the other 2 original disks, the disks having no problems. Can I at least go back to the situation before the cache clearance? As the original post demonstrates, I then had 4 correctly registered PVs (plus 2 fakes).
Note: vgreduce will not work untill the metadata are corrected.
 
Old 08-21-2018, 02:06 PM   #4
roelvdh
LQ Newbie
 
Registered: Aug 2018
Location: Netherlands
Distribution: ClearOS
Posts: 26

Original Poster
Rep: Reputation: Disabled
I tried to restore the metadata from backup, but that failed as well.
I'm pretty sure the backup at /etc/lvm/backup/AV is exactly right:

PHP Code:
# Generated by LVM2 version 2.02.177(2)-RHEL7 (2018-01-22): Sat Aug 18 17:36:16 2018

contents "Text Format Volume Group"
version 1

description 
"Created *after* executing 'vgreduce --removemissing AV'"

creation_host "san2.xxx.com"    # Linux san2.xxx.com 3.10.0-862.11.6.v7.x86_64 #1 SMP Wed Aug 15 20:03:47 MDT 2018 x86_64
creation_time 1534606576    # Sat Aug 18 17:36:16 2018

AV {
    
id "JjSj9f-FV5T-GF1P-TBPS-JTwq-qtb6-VJ0Ljv"
    
seqno 14
    format 
"lvm2"            # informational
    
status = ["RESIZEABLE""READ""WRITE"]
    
flags = []
    
extent_size 8192        # 4 Megabytes
    
max_lv 0
    max_pv 
0
    metadata_copies 
0

    physical_volumes 
{

        
pv0 {
            
id "w7WVft-LUpq-sjEc-IJFm-c41t-Qv9W-PYzIlM"
            
device "/dev/sde1"    # Hint only

            
status = ["ALLOCATABLE"]
            
flags = []
            
dev_size 7814037095    # 3,63869 Terabytes
            
pe_start 65535
            pe_count 
953853    # 3,63866 Terabytes
        
}

        
pv1 {
            
id "JAyBXE-aZrQ-dVUO-UB0W-PQuT-GTrT-IEYT2w"
            
device "/dev/sdf1"    # Hint only

            
status = ["ALLOCATABLE"]
            
flags = []
            
dev_size 5860533095    # 2,72902 Terabytes
            
pe_start 65535
            pe_count 
715389    # 2,72899 Terabytes
        
}

        
pv2 {
            
id "8C8y2a-GXmN-tPNx-Jha2-Rea1-cqav-AfyfNm"
            
device "/dev/sdi1"    # Hint only

            
status = ["ALLOCATABLE"]
            
flags = []
            
dev_size 7814037064    # 3,63869 Terabytes
            
pe_start 65528
            pe_count 
953853    # 3,63866 Terabytes
        
}

        
pv3 {
            
id "JTdK18-uuaq-8U4t-RQ2I-etKd-jUF5-03xIjk"
            
device "/dev/sdj1"    # Hint only

            
status = ["ALLOCATABLE"]
            
flags = []
            
dev_size 7814037064    # 3,63869 Terabytes
            
pe_start 65528
            pe_count 
953853    # 3,63866 Terabytes
        

But I can't restore this file. The 2 new devices /dev/sdi and /dev/sdj seem to have lost their LVM info. I tried to pvcreate these devices again but that failed. How do I correctly restore the backup file?
 
Old 08-21-2018, 02:32 PM   #5
smallpond
Senior Member
 
Registered: Feb 2011
Location: Massachusetts, USA
Distribution: Fedora
Posts: 4,140

Rep: Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263
First I think you need to figure out what caused the duplicate uuid message. Do you have multiple paths to the drives? Did you not remove the failed drives after ddrescue? Or is it possible the wrong drives were copied?

Check for multiple paths with: multipath -ll

Check the list of all drives with: lsblk -a
 
Old 08-21-2018, 03:20 PM   #6
roelvdh
LQ Newbie
 
Registered: Aug 2018
Location: Netherlands
Distribution: ClearOS
Posts: 26

Original Poster
Rep: Reputation: Disabled
Thank you. I do not use multipath. lsblk shows exactly correct info:

PHP Code:
[root@san2 ~]# lsblk -a
NAME             MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                8
:0    0  14,9G  0 disk
&#9500;─sda1             8:1    0   200M  0 part /boot/efi
&#9500;─sda2             8:2    0     1G  0 part /boot
&#9492;─sda3             8:3    0  13,7G  0 part
  
&#9500;─clearos-root 253:0    0  12,2G  0 lvm  /
  
&#9492;─clearos-swap 253:1    0   1,5G  0 lvm  [SWAP]
sde                8:64   0   3,7T  0 disk
&#9492;─sde1             8:65   0   3,7T  0 part
sdf                8:80   0   2,7T  0 disk
&#9492;─sdf1             8:81   0   2,7T  0 part
sdi                8:128  0   4,6T  0 disk
&#9492;─sdi1             8:129  0   4,6T  0 part
sdj                8:144  0   4,6T  0 disk
&#9492;─sdj1             8:145  0   4,6T  0 part 
I was thinking of entirely removing the VG, rebuild it and restore the last backup file. Would that be safe, i.e. preserve all data?
Something like:

vgremove --force AV
pvcreate --uuid 8C8y2a-GXmN-tPNx-Jha2-Rea1-cqav-AfyfNm /dev/sdi1
pvcreate --uuid JTdK18-uuaq-8U4t-RQ2I-etKd-jUF5-03xIjk /dev/sdj1
pvscan (to check if all 4 devices are there)
vgcfgrestore --file <last backup file> AV

Could this help resolve the problem?
 
Old 08-22-2018, 08:39 AM   #7
smallpond
Senior Member
 
Registered: Feb 2011
Location: Massachusetts, USA
Distribution: Fedora
Posts: 4,140

Rep: Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263
If you need to preserve your data then my best advice is don't do anything to those disks except copy them to new disks and test recovery on those.
 
Old 08-22-2018, 09:52 AM   #8
roelvdh
LQ Newbie
 
Registered: Aug 2018
Location: Netherlands
Distribution: ClearOS
Posts: 26

Original Poster
Rep: Reputation: Disabled
I understand, but I would still face the same problem as the metadata would still be mixed up. LVM doesn't let me use any LVM command before this is rectified, and it won't even let me rectify. I really need to do something more definitive with the metadata.
I do have the correct number of devices connected as lsblk shows, and I do have a correct lvm config file that worked in the beginning. Isn't there a way to get rid of all caches etc (LVM won't let me run pvscan --cache) and just use the bare config file that used to work before LVM mixed it all up? Thanks for trying to help me out.
 
Old 08-22-2018, 01:05 PM   #9
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: Rocky Linux
Posts: 4,776

Rep: Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212
In case you did not know, those files in /etc/lvm/archive and /etc/lvm/backup are exactly that, archive and backup. They are not controlling. The entire state of the volume group is recorded on each PV in the volume group in its LVM header. The file in /etc/lvm/backup is just a formatted and commented version of the ASCII text that begins in the 9th 512-byte sector of that header.

It's not clear just how you got into your current state, but you can unconditionally clear all your LVM headers by running "dd if=/dev/zero bs=1k count=10 of=/dev/{xxxx}" on each of your physical volumes. You will then need to restore those PVs by running pvcreate with the "--uuid" and "--restorefile" options using your known good configuration file as the data source. That should get you into a state where you can do a successful vgcfgrestore.
 
1 members found this post helpful.
Old 08-23-2018, 02:23 AM   #10
roelvdh
LQ Newbie
 
Registered: Aug 2018
Location: Netherlands
Distribution: ClearOS
Posts: 26

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by rknichols View Post
In case you did not know, those files in /etc/lvm/archive and /etc/lvm/backup are exactly that, archive and backup. They are not controlling. The entire state of the volume group is recorded on each PV in the volume group in its LVM header. The file in /etc/lvm/backup is just a formatted and commented version of the ASCII text that begins in the 9th 512-byte sector of that header.

It's not clear just how you got into your current state, but you can unconditionally clear all your LVM headers by running "dd if=/dev/zero bs=1k count=10 of=/dev/{xxxx}" on each of your physical volumes. You will then need to restore those PVs by running pvcreate with the "--uuid" and "--restorefile" options using your known good configuration file as the data source. That should get you into a state where you can do a successful vgcfgrestore.
Many thanks for the explanation. This clarified my situation and led me to successfully solve the problem.
 
Old 08-23-2018, 03:23 AM   #11
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,125

Rep: Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120
I was hoping that the "wipe" option of (newer) fdisk would erase the LVM meta-data, but a quick test proved disappointing, so we'll have to keep using dd for the foreseeable future.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Impossible to remove mysql on 16.04 - constant errors preventing system-wide upgrade bulgin Ubuntu 2 05-21-2018 08:18 PM
[SOLVED] Debian custom encrypted LVM install - Impossible to achieve Amarildo Linux - Newbie 8 03-24-2015 04:19 PM
Used vgremove instead of pvremove milomak Linux - Software 4 10-30-2013 02:40 PM
LVM : pvremove and now volume group not found sierramike Linux - General 2 08-12-2011 09:14 AM
Konqueror history impossible to remove. pujolasdf Linux - Software 5 02-16-2005 04:05 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > CentOS

All times are GMT -5. The time now is 06:12 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration