LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Server (http://www.linuxquestions.org/questions/linux-server-73/)
-   -   Problem with LVM mirror and failed PV, can't recover -- Debian Squeeze lvm 2.02.66-5 (http://www.linuxquestions.org/questions/linux-server-73/problem-with-lvm-mirror-and-failed-pv-cant-recover-debian-squeeze-lvm-2-02-66-5-a-897336/)

Linux Ninja 08-14-2011 11:43 PM

Problem with LVM mirror and failed PV, can't recover -- Debian Squeeze lvm 2.02.66-5
 
I have a Logical volume that is mirrored, albeit not ideally, but one of the PVs has failed. No command that I have come up with will complete, due to the missing PV.

Here's an output of `lvs -a -v -o+devices`:

Code:

# lvs -a -v -o+devices
    Finding all logical volumes
  /dev/dm-16: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-17: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-18: read failed after 0 of 4096 at 0: Input/output error
    Wiping cache of LVM-capable devices
  /dev/dm-16: read failed after 0 of 4096 at 16710041600: Input/output error
  /dev/dm-16: read failed after 0 of 4096 at 16710098944: Input/output error
    /dev/dm-16: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-16: read failed after 0 of 4096 at 4096: Input/output error
    /dev/dm-16: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-17: read failed after 0 of 4096 at 52613283840: Input/output error
  /dev/dm-17: read failed after 0 of 4096 at 52613341184: Input/output error
    /dev/dm-17: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-17: read failed after 0 of 4096 at 4096: Input/output error
    /dev/dm-17: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-18: read failed after 0 of 4096 at 42949607424: Input/output error
  /dev/dm-18: read failed after 0 of 4096 at 42949664768: Input/output error
    /dev/dm-18: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-18: read failed after 0 of 4096 at 4096: Input/output error
    /dev/dm-18: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-21: read failed after 0 of 4096 at 503316414464: Input/output error
  /dev/dm-21: read failed after 0 of 4096 at 503316471808: Input/output error
  Couldn't find device with uuid K7Xklz-OruW-4kDk-1GbN-tEPz-xASw-ZoWCqB.
    /dev/dm-16: read failed after 0 of 4096 at 16710041600: Input/output error
    /dev/dm-16: read failed after 0 of 4096 at 16710098944: Input/output error
    /dev/dm-16: read failed after 0 of 4096 at 0: Input/output error
    /dev/dm-16: read failed after 0 of 4096 at 4096: Input/output error
    /dev/dm-16: read failed after 0 of 4096 at 0: Input/output error
    /dev/dm-17: read failed after 0 of 4096 at 52613283840: Input/output error
    /dev/dm-17: read failed after 0 of 4096 at 52613341184: Input/output error
    /dev/dm-17: read failed after 0 of 4096 at 0: Input/output error
    /dev/dm-17: read failed after 0 of 4096 at 4096: Input/output error
    /dev/dm-17: read failed after 0 of 4096 at 0: Input/output error
    /dev/dm-18: read failed after 0 of 4096 at 42949607424: Input/output error
    /dev/dm-18: read failed after 0 of 4096 at 42949664768: Input/output error
    /dev/dm-18: read failed after 0 of 4096 at 0: Input/output error
    /dev/dm-18: read failed after 0 of 4096 at 4096: Input/output error
    /dev/dm-18: read failed after 0 of 4096 at 0: Input/output error
    /dev/dm-21: read failed after 0 of 4096 at 503316414464: Input/output error
    /dev/dm-21: read failed after 0 of 4096 at 503316471808: Input/output error
    Couldn't find device with uuid K7Xklz-OruW-4kDk-1GbN-tEPz-xASw-ZoWCqB.
    There are 1 physical volumes missing.
  LV                    VG        #Seg Attr  LSize  Maj Min KMaj KMin Origin Snap%  Move Copy%  Log Convert LV UUID                                Devices
  LVhome                VolGroup00    1 mwi-a- 468.75g  -1  -1 254  21                      59.63            JXflDp-bh8q-TLQ7-wUr9-2vDV-E151-RrwuvW LVhome_mimage_0(0),LVhome_mimage_1(0)
  [LVhome_mimage_0]    VolGroup00    6 Iwi-ao 468.75g  -1  -1 254  19                                        Ydd8BX-nH0B-aObV-n4rR-4D3c-HPw5-T05FQC /dev/sdc1(19076)
  [LVhome_mimage_0]    VolGroup00    6 Iwi-ao 468.75g  -1  -1 254  19                                        Ydd8BX-nH0B-aObV-n4rR-4D3c-HPw5-T05FQC unknown device(25052)
  [LVhome_mimage_0]    VolGroup00    6 Iwi-ao 468.75g  -1  -1 254  19                                        Ydd8BX-nH0B-aObV-n4rR-4D3c-HPw5-T05FQC unknown device(28240)
  [LVhome_mimage_0]    VolGroup00    6 Iwi-ao 468.75g  -1  -1 254  19                                        Ydd8BX-nH0B-aObV-n4rR-4D3c-HPw5-T05FQC unknown device(14400)
  [LVhome_mimage_0]    VolGroup00    6 Iwi-ao 468.75g  -1  -1 254  19                                        Ydd8BX-nH0B-aObV-n4rR-4D3c-HPw5-T05FQC /dev/sda2(26906)
  [LVhome_mimage_0]    VolGroup00    6 Iwi-ao 468.75g  -1  -1 254  19                                        Ydd8BX-nH0B-aObV-n4rR-4D3c-HPw5-T05FQC /dev/sda2(27556)
  [LVhome_mimage_1]    VolGroup00    4 Iwi-ao 468.75g  -1  -1 254  20                                        3f4tfM-lOWt-ofA0-ruln-DGbI-cs4l-sAaOqM /dev/sda2(11090)
  [LVhome_mimage_1]    VolGroup00    4 Iwi-ao 468.75g  -1  -1 254  20                                        3f4tfM-lOWt-ofA0-ruln-DGbI-cs4l-sAaOqM /dev/sda2(24058)
  [LVhome_mimage_1]    VolGroup00    4 Iwi-ao 468.75g  -1  -1 254  20                                        3f4tfM-lOWt-ofA0-ruln-DGbI-cs4l-sAaOqM /dev/sdc1(25780)
  [LVhome_mimage_1]    VolGroup00    4 Iwi-ao 468.75g  -1  -1 254  20                                        3f4tfM-lOWt-ofA0-ruln-DGbI-cs4l-sAaOqM /dev/sdc1(10528)

I'm unable to recover from this, and ALL of my critical data is in this LV. I thought using a Mirror would prevent a drive failure from causing me to lose data... but it seems no configuration, even ones intended to allow you to proceed with a missing mirrored drive, behaves in an expected way.

I've tried splitting the mirror, repairing, converting to linear... nothing works. Everything tells me that it won't make any change to the VG with missing PVs. The copy% you see above is when I run lvchange -ay --partial VolGroup00/LVhome - this seems to be the only way to get lvm to do anything with this LV. When I mount it, I get a lot of garbage in the ls. Really? Two copies, and I've got garbage?

Has anyone else run into this? I suspect part of the problem being that mirror 1 lies on two separate PVs (as well as mirror 0). I should have access to all of my data. Not even fsck will run without an error that scares the bejeezus out of me.

As a side note, I had other LVs that were mirrored, with one leg on the failed PV, and I had no problem recovering them. But, they had each leg entirely on a single PV. The LV I have a problem with has been resized a few times to add space, since this is the /home mount. Apparently, the resize operation didn't keep the legs split across the PVs like it should. Good thing I didn't lose the PV that mirror 1 sits on, since mirror 0 also straddles it.

Code:

# e2fsck -p -f /dev/mapper/VolGroup00-LVhome
Error reading block 64211540 (Attempt to read block from filesystem resulted in short read) while reading indirect blocks of inode 426151.

/dev/mapper/VolGroup00-LVhome: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
        (i.e., without -a or -p options)

Code:

# lvconvert --splitmirrors 1 --name homecopy VolGroup00/LVhome
  /dev/dm-16: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-17: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-18: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-16: read failed after 0 of 4096 at 16710041600: Input/output error
  /dev/dm-16: read failed after 0 of 4096 at 16710098944: Input/output error
  /dev/dm-16: read failed after 0 of 4096 at 4096: Input/output error
  /dev/dm-17: read failed after 0 of 4096 at 52613283840: Input/output error
  /dev/dm-17: read failed after 0 of 4096 at 52613341184: Input/output error
  /dev/dm-17: read failed after 0 of 4096 at 4096: Input/output error
  /dev/dm-18: read failed after 0 of 4096 at 42949607424: Input/output error
  /dev/dm-18: read failed after 0 of 4096 at 42949664768: Input/output error
  /dev/dm-18: read failed after 0 of 4096 at 4096: Input/output error
  /dev/dm-21: read failed after 0 of 4096 at 503316414464: Input/output error
  /dev/dm-21: read failed after 0 of 4096 at 503316471808: Input/output error
  Couldn't find device with uuid K7Xklz-OruW-4kDk-1GbN-tEPz-xASw-ZoWCqB.
  Cannot change VG VolGroup00 while PVs are missing.
  Consider vgreduce --removemissing.

If I could just mount the intact mirror by itself, I'd be in good shape. I can't even add another PV to the VG in this situation to give me more space.

Speedy G 06-01-2017 03:51 PM

6 disk LVM Mirror - 1 Dead Disk - Need to recover data
 
Just like the previous entry says, one of my disks died in a 2-way mirror of 6 disks. I need to recover the data, that's why I did a mirror setup. System is Centos 6.4.

lvs -a -o +devices
/dev/sdd: read failed after 0 of 4096 at 0: Input/output error
/dev/sdd: read failed after 0 of 4096 at 2000398843904: Input/output error
/dev/sdd: read failed after 0 of 4096 at 2000398925824: Input/output error
/dev/sdd: read failed after 0 of 4096 at 4096: Input/output error
/dev/sdd1: read failed after 0 of 1024 at 2000396222464: Input/output error
/dev/sdd1: read failed after 0 of 1024 at 2000396279808: Input/output error
/dev/sdd1: read failed after 0 of 1024 at 0: Input/output error
/dev/sdd1: read failed after 0 of 1024 at 4096: Input/output error
/dev/sdd1: read failed after 0 of 2048 at 0: Input/output error
Couldn't find device with uuid a4ROgi-JP22-0llk-G39I-n9BP-G2cF-WCO4Yk.
LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert Devices
home vg_data mwi-a-m-p 4.50t 100.00 home_mimage_0(0),home_mimage_1(0)
[home_mimage_0] vg_data Iwi-aom-- 4.50t /dev/sda2(0)
[home_mimage_0] vg_data Iwi-aom-- 4.50t /dev/sdc1(0)
[home_mimage_0] vg_data Iwi-aom-- 4.50t /dev/sde1(0)
[home_mimage_1] vg_data Iwi-aom-p 4.50t /dev/sdb2(0)
[home_mimage_1] vg_data Iwi-aom-p 4.50t unknown device(0)
[home_mimage_1] vg_data Iwi-aom-p 4.50t /dev/sdf1(0)
LogVol00 vg_system -wi-ao--- 97.65g /dev/sda3(0)
LogVol00 vg_system -wi-ao--- 97.65g /dev/sdb3(0)



Nothing works:

lvconvert --splitmirrors 1 --name backup2 vg_data/home
Couldn't find device with uuid a4ROgi-JP22-0llk-G39I-n9BP-G2cF-WCO4Yk.
Cannot change VG vg_data while PVs are missing.
Consider vgreduce --removemissing.

Speedy G 06-02-2017 04:50 PM

I fixed the problem. What I think happend is as follows:

1) LVM detected the drive was bad and switched to linear mode. I had a 3 segment 2-way mirror. The middle drive of one of the mirrors died.
2) The whole LV went into R/O mode and somehow the state of the drive didn't change, so it showed that the mirror was still there but I couldn't alter it.
3) I tried all lvconvert and vgreduce commands that I could think of but nothing worked since it was in R/O mode.
4) I rebooted after removing the LV from fstab (it was /home that failed). The LV was recognized and now it was mountable and although it threw the error that a PV was missing, I was able to vgreduce --removemissing.
5) Now I have a 5 drive LV that doesn't say whether the mirror is still there, but all the data is accessible. Now I'll have to redo the mirroring and I should be done.

Speedy G


All times are GMT -5. The time now is 07:05 PM.