LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Server (http://www.linuxquestions.org/questions/linux-server-73/)
-   -   Problem with LVM mirror and failed PV, can't recover -- Debian Squeeze lvm 2.02.66-5 (http://www.linuxquestions.org/questions/linux-server-73/problem-with-lvm-mirror-and-failed-pv-can%27t-recover-debian-squeeze-lvm-2-02-66-5-a-897336/)

Linux Ninja 08-14-2011 11:43 PM

Problem with LVM mirror and failed PV, can't recover -- Debian Squeeze lvm 2.02.66-5
 
I have a Logical volume that is mirrored, albeit not ideally, but one of the PVs has failed. No command that I have come up with will complete, due to the missing PV.

Here's an output of `lvs -a -v -o+devices`:

Code:

# lvs -a -v -o+devices
    Finding all logical volumes
  /dev/dm-16: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-17: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-18: read failed after 0 of 4096 at 0: Input/output error
    Wiping cache of LVM-capable devices
  /dev/dm-16: read failed after 0 of 4096 at 16710041600: Input/output error
  /dev/dm-16: read failed after 0 of 4096 at 16710098944: Input/output error
    /dev/dm-16: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-16: read failed after 0 of 4096 at 4096: Input/output error
    /dev/dm-16: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-17: read failed after 0 of 4096 at 52613283840: Input/output error
  /dev/dm-17: read failed after 0 of 4096 at 52613341184: Input/output error
    /dev/dm-17: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-17: read failed after 0 of 4096 at 4096: Input/output error
    /dev/dm-17: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-18: read failed after 0 of 4096 at 42949607424: Input/output error
  /dev/dm-18: read failed after 0 of 4096 at 42949664768: Input/output error
    /dev/dm-18: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-18: read failed after 0 of 4096 at 4096: Input/output error
    /dev/dm-18: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-21: read failed after 0 of 4096 at 503316414464: Input/output error
  /dev/dm-21: read failed after 0 of 4096 at 503316471808: Input/output error
  Couldn't find device with uuid K7Xklz-OruW-4kDk-1GbN-tEPz-xASw-ZoWCqB.
    /dev/dm-16: read failed after 0 of 4096 at 16710041600: Input/output error
    /dev/dm-16: read failed after 0 of 4096 at 16710098944: Input/output error
    /dev/dm-16: read failed after 0 of 4096 at 0: Input/output error
    /dev/dm-16: read failed after 0 of 4096 at 4096: Input/output error
    /dev/dm-16: read failed after 0 of 4096 at 0: Input/output error
    /dev/dm-17: read failed after 0 of 4096 at 52613283840: Input/output error
    /dev/dm-17: read failed after 0 of 4096 at 52613341184: Input/output error
    /dev/dm-17: read failed after 0 of 4096 at 0: Input/output error
    /dev/dm-17: read failed after 0 of 4096 at 4096: Input/output error
    /dev/dm-17: read failed after 0 of 4096 at 0: Input/output error
    /dev/dm-18: read failed after 0 of 4096 at 42949607424: Input/output error
    /dev/dm-18: read failed after 0 of 4096 at 42949664768: Input/output error
    /dev/dm-18: read failed after 0 of 4096 at 0: Input/output error
    /dev/dm-18: read failed after 0 of 4096 at 4096: Input/output error
    /dev/dm-18: read failed after 0 of 4096 at 0: Input/output error
    /dev/dm-21: read failed after 0 of 4096 at 503316414464: Input/output error
    /dev/dm-21: read failed after 0 of 4096 at 503316471808: Input/output error
    Couldn't find device with uuid K7Xklz-OruW-4kDk-1GbN-tEPz-xASw-ZoWCqB.
    There are 1 physical volumes missing.
  LV                    VG        #Seg Attr  LSize  Maj Min KMaj KMin Origin Snap%  Move Copy%  Log Convert LV UUID                                Devices
  LVhome                VolGroup00    1 mwi-a- 468.75g  -1  -1 254  21                      59.63            JXflDp-bh8q-TLQ7-wUr9-2vDV-E151-RrwuvW LVhome_mimage_0(0),LVhome_mimage_1(0)
  [LVhome_mimage_0]    VolGroup00    6 Iwi-ao 468.75g  -1  -1 254  19                                        Ydd8BX-nH0B-aObV-n4rR-4D3c-HPw5-T05FQC /dev/sdc1(19076)
  [LVhome_mimage_0]    VolGroup00    6 Iwi-ao 468.75g  -1  -1 254  19                                        Ydd8BX-nH0B-aObV-n4rR-4D3c-HPw5-T05FQC unknown device(25052)
  [LVhome_mimage_0]    VolGroup00    6 Iwi-ao 468.75g  -1  -1 254  19                                        Ydd8BX-nH0B-aObV-n4rR-4D3c-HPw5-T05FQC unknown device(28240)
  [LVhome_mimage_0]    VolGroup00    6 Iwi-ao 468.75g  -1  -1 254  19                                        Ydd8BX-nH0B-aObV-n4rR-4D3c-HPw5-T05FQC unknown device(14400)
  [LVhome_mimage_0]    VolGroup00    6 Iwi-ao 468.75g  -1  -1 254  19                                        Ydd8BX-nH0B-aObV-n4rR-4D3c-HPw5-T05FQC /dev/sda2(26906)
  [LVhome_mimage_0]    VolGroup00    6 Iwi-ao 468.75g  -1  -1 254  19                                        Ydd8BX-nH0B-aObV-n4rR-4D3c-HPw5-T05FQC /dev/sda2(27556)
  [LVhome_mimage_1]    VolGroup00    4 Iwi-ao 468.75g  -1  -1 254  20                                        3f4tfM-lOWt-ofA0-ruln-DGbI-cs4l-sAaOqM /dev/sda2(11090)
  [LVhome_mimage_1]    VolGroup00    4 Iwi-ao 468.75g  -1  -1 254  20                                        3f4tfM-lOWt-ofA0-ruln-DGbI-cs4l-sAaOqM /dev/sda2(24058)
  [LVhome_mimage_1]    VolGroup00    4 Iwi-ao 468.75g  -1  -1 254  20                                        3f4tfM-lOWt-ofA0-ruln-DGbI-cs4l-sAaOqM /dev/sdc1(25780)
  [LVhome_mimage_1]    VolGroup00    4 Iwi-ao 468.75g  -1  -1 254  20                                        3f4tfM-lOWt-ofA0-ruln-DGbI-cs4l-sAaOqM /dev/sdc1(10528)

I'm unable to recover from this, and ALL of my critical data is in this LV. I thought using a Mirror would prevent a drive failure from causing me to lose data... but it seems no configuration, even ones intended to allow you to proceed with a missing mirrored drive, behaves in an expected way.

I've tried splitting the mirror, repairing, converting to linear... nothing works. Everything tells me that it won't make any change to the VG with missing PVs. The copy% you see above is when I run lvchange -ay --partial VolGroup00/LVhome - this seems to be the only way to get lvm to do anything with this LV. When I mount it, I get a lot of garbage in the ls. Really? Two copies, and I've got garbage?

Has anyone else run into this? I suspect part of the problem being that mirror 1 lies on two separate PVs (as well as mirror 0). I should have access to all of my data. Not even fsck will run without an error that scares the bejeezus out of me.

As a side note, I had other LVs that were mirrored, with one leg on the failed PV, and I had no problem recovering them. But, they had each leg entirely on a single PV. The LV I have a problem with has been resized a few times to add space, since this is the /home mount. Apparently, the resize operation didn't keep the legs split across the PVs like it should. Good thing I didn't lose the PV that mirror 1 sits on, since mirror 0 also straddles it.

Code:

# e2fsck -p -f /dev/mapper/VolGroup00-LVhome
Error reading block 64211540 (Attempt to read block from filesystem resulted in short read) while reading indirect blocks of inode 426151.

/dev/mapper/VolGroup00-LVhome: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
        (i.e., without -a or -p options)

Code:

# lvconvert --splitmirrors 1 --name homecopy VolGroup00/LVhome
  /dev/dm-16: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-17: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-18: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-16: read failed after 0 of 4096 at 16710041600: Input/output error
  /dev/dm-16: read failed after 0 of 4096 at 16710098944: Input/output error
  /dev/dm-16: read failed after 0 of 4096 at 4096: Input/output error
  /dev/dm-17: read failed after 0 of 4096 at 52613283840: Input/output error
  /dev/dm-17: read failed after 0 of 4096 at 52613341184: Input/output error
  /dev/dm-17: read failed after 0 of 4096 at 4096: Input/output error
  /dev/dm-18: read failed after 0 of 4096 at 42949607424: Input/output error
  /dev/dm-18: read failed after 0 of 4096 at 42949664768: Input/output error
  /dev/dm-18: read failed after 0 of 4096 at 4096: Input/output error
  /dev/dm-21: read failed after 0 of 4096 at 503316414464: Input/output error
  /dev/dm-21: read failed after 0 of 4096 at 503316471808: Input/output error
  Couldn't find device with uuid K7Xklz-OruW-4kDk-1GbN-tEPz-xASw-ZoWCqB.
  Cannot change VG VolGroup00 while PVs are missing.
  Consider vgreduce --removemissing.

If I could just mount the intact mirror by itself, I'd be in good shape. I can't even add another PV to the VG in this situation to give me more space.


All times are GMT -5. The time now is 09:57 PM.