Problem with LVM. "duplicate PVs were found"

mathrixx · 04-04-2020, 04:00 PM

Hi guys!

I have no idea how did it happen, but it did. Today I was upgrading packages, and during that, I saw that there are problems with LVM. I've checked it out by doing the following, but have no idea how to fix it.

Can you help me with that?

Code:

# pvdisplay
  WARNING: Not using lvmetad because duplicate PVs were found.
  WARNING: Use multipath or vgimportclone to resolve duplicate PVs?
  WARNING: After duplicates are resolved, run "pvscan --cache" to enable lvmetad.
  WARNING: PV rGZ9x1-meyt-4QyW-0N1Q-uqvx-1bxa-1YZSxv on /dev/sda5 was already found on /dev/mapper/isw_cbhjibbge_backup5.
  WARNING: PV rGZ9x1-meyt-4QyW-0N1Q-uqvx-1bxa-1YZSxv prefers device /dev/mapper/isw_cbhjibbge_backup5 because device is used by LV.
  --- Physical volume ---
  PV Name               /dev/mapper/isw_cbhjibbge_backup5
  VG Name               ubuntu-vg
  PV Size               <465,52 GiB / not usable 4,00 MiB
  Allocatable           yes
  PE Size               4,00 MiB
  Total PE              119172
  Free PE               11
  Allocated PE          119161
  PV UUID               rGZ9x1-meyt-4QyW-0N1Q-uqvx-1bxa-1YZSxv

# lsblk -f
NAME                      FSTYPE          LABEL UUID                                   MOUNTPOINT
sda                       isw_raid_member
├─sda1                    ext2                  b794e92b-dbe1-4d2c-87b1-700bc1c3b308
├─sda2                    isw_raid_member
├─sda5                    LVM2_member           rGZ9x1-meyt-4QyW-0N1Q-uqvx-1bxa-1YZSxv
└─isw_cbhjibbge_backup
  ├─isw_cbhjibbge_backup1 ext2                  b794e92b-dbe1-4d2c-87b1-700bc1c3b308   /boot
  ├─isw_cbhjibbge_backup2
  └─isw_cbhjibbge_backup5 LVM2_member           rGZ9x1-meyt-4QyW-0N1Q-uqvx-1bxa-1YZSxv
    ├─ubuntu--vg-root     ext4                  b6750ef2-c8bb-4f2b-aee8-2bed858b9b3d   /
    └─ubuntu--vg-swap_1   swap                  f922c46f-98a2-4257-85ed-81b29f0db0b6   [SWAP]
sdb                       isw_raid_member
└─isw_cbhjibbge_backup
  ├─isw_cbhjibbge_backup1 ext2                  b794e92b-dbe1-4d2c-87b1-700bc1c3b308   /boot
  ├─isw_cbhjibbge_backup2
  └─isw_cbhjibbge_backup5 LVM2_member           rGZ9x1-meyt-4QyW-0N1Q-uqvx-1bxa-1YZSxv
    ├─ubuntu--vg-root     ext4                  b6750ef2-c8bb-4f2b-aee8-2bed858b9b3d   /
    └─ubuntu--vg-swap_1   swap                  f922c46f-98a2-4257-85ed-81b29f0db0b6   [SWAP]
sr0

rknichols · 04-04-2020, 09:45 PM

As the warning message suggests, use vgimportclone to rename the conflicting volume group and give its volumes unique UUIDs. Making backups by copying entire devices bit-for-bit causes problems because everything, including names, labels, and UUIDs, gets copied, leaving you with duplicates. You have a lot of duplicated UUIDs in that list. That's a bad situation that should be corrected. As it stands, you can find you are not actually using the volumes and filesystems that you think you are.

See the manpage for vgimportclone.

syg00 · 04-04-2020, 10:27 PM

You sure about that - the OP did say they had been happily using the system.?.
To me that looks like a Intel on-board RAID set that has broken for some reason. I might be inclined to see if there have been any failure messages. Strange that the disk layout is different - might indicate /dev/sdb might have an issue.

rknichols · 04-05-2020, 07:55 AM

Could be. I would have hoped that the OP would have noticed that RAID was running in a degraded mode and fixed that before trying to upgrade packages.

mathrixx · 04-05-2020, 12:00 PM

Ok, so what I did today is running a long SMART test on both disks. Results are below. Only one parameter whis doesn't look nice is Command_Timeout on /dev/sda

Code:

root@BackupSRV:~# smartctl -a -d ata /dev/sda
smartctl 6.6 2016-05-31 r4324 [i686-linux-4.15.0-91-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.14 (AF)
Device Model:     ST500DM002-1BD142
Serial Number:    Z6E9DNEY
LU WWN Device Id: 5 000c50 078f734df
Firmware Version: KC45
User Capacity:    500 107 862 016 bytes [500 GB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 1.5 Gb/s)
Local Time is:    Sun Apr  5 18:42:20 2020 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  600) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        (  82) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x303f) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   110   099   006    Pre-fail  Always       -       28969888
  3 Spin_Up_Time            0x0003   100   100   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       446
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   081   060   030    Pre-fail  Always       -       152138602
  9 Power_On_Hours          0x0032   058   058   000    Old_age   Always       -       37134
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       459
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   099   000    Old_age   Always       -       1 1 1
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   065   055   045    Old_age   Always       -       35 (Min/Max 32/40)
194 Temperature_Celsius     0x0022   035   045   000    Old_age   Always       -       35 (0 17 0 0 0)
195 Hardware_ECC_Recovered  0x001a   035   012   000    Old_age   Always       -       28969888
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       37121h+04m+26.505s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       684837372
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       3398507033

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     37129         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.



root@BackupSRV:~# smartctl -a -d ata /dev/sdb
smartctl 6.6 2016-05-31 r4324 [i686-linux-4.15.0-91-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.14 (AF)
Device Model:     ST500DM002-1BD142
Serial Number:    W3TCR73Y
LU WWN Device Id: 5 000c50 07c3b50a1
Firmware Version: KC48
User Capacity:    500 107 862 016 bytes [500 GB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 1.5 Gb/s)
Local Time is:    Sun Apr  5 18:42:50 2020 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  609) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        (  82) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x303f) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   108   099   006    Pre-fail  Always       -       19081728
  3 Spin_Up_Time            0x0003   100   099   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       455
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   076   060   030    Pre-fail  Always       -       13010797834
  9 Power_On_Hours          0x0032   058   058   000    Old_age   Always       -       37141
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       464
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0 0 0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   066   053   045    Old_age   Always       -       34 (Min/Max 32/38)
194 Temperature_Celsius     0x0022   034   047   000    Old_age   Always       -       34 (0 18 0 0 0)
195 Hardware_ECC_Recovered  0x001a   012   012   000    Old_age   Always       -       19081728
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       34674h+29m+28.670s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       1322122211
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       2020253685

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     37138         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

If by saying OP you mean 'me', then you are right. I did not check RAID status before I did packets upgrade. I don't even remember am I using there onboard RAID, or lvm only. I've built that simple samba server about 5 years ago and I'm not looking out there very often.

mathrixx · 04-05-2020, 01:05 PM

I have trouble to figure out how to use vgimportclone to rename the conflicting volume group and give its volumes unique UUIDs.
I just don't want to risk when I don't understand the synopsis as it does not stand straight for an option to change UUID of a volume group.

Can you help me with that and tell me how exactly should I run vgimportclone with clarification why?

rknichols · 04-05-2020, 10:27 PM

It all depends on whether you want to use the disks as a RAID 1 array (in which case they should be duplicates) or use them as independent disks (in which case the UUIDs definitely should not be duplicated). The drives seem to have become JBOD now, and the system does not like seeing the duplicates. If you want RAID, you will have to work out how to get the array back in sync, making it appear as a single device and not two separate drives.