LinuxQuestions.org
Register a domain and help support LQ
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 08-14-2011, 11:43 PM   #1
Linux Ninja
LQ Newbie
 
Registered: Aug 2011
Location: Atlanta, GA
Distribution: Debian
Posts: 1

Rep: Reputation: Disabled
Problem with LVM mirror and failed PV, can't recover -- Debian Squeeze lvm 2.02.66-5


I have a Logical volume that is mirrored, albeit not ideally, but one of the PVs has failed. No command that I have come up with will complete, due to the missing PV.

Here's an output of `lvs -a -v -o+devices`:

Code:
# lvs -a -v -o+devices
    Finding all logical volumes
  /dev/dm-16: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-17: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-18: read failed after 0 of 4096 at 0: Input/output error
    Wiping cache of LVM-capable devices
  /dev/dm-16: read failed after 0 of 4096 at 16710041600: Input/output error
  /dev/dm-16: read failed after 0 of 4096 at 16710098944: Input/output error
    /dev/dm-16: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-16: read failed after 0 of 4096 at 4096: Input/output error
    /dev/dm-16: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-17: read failed after 0 of 4096 at 52613283840: Input/output error
  /dev/dm-17: read failed after 0 of 4096 at 52613341184: Input/output error
    /dev/dm-17: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-17: read failed after 0 of 4096 at 4096: Input/output error
    /dev/dm-17: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-18: read failed after 0 of 4096 at 42949607424: Input/output error
  /dev/dm-18: read failed after 0 of 4096 at 42949664768: Input/output error
    /dev/dm-18: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-18: read failed after 0 of 4096 at 4096: Input/output error
    /dev/dm-18: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-21: read failed after 0 of 4096 at 503316414464: Input/output error
  /dev/dm-21: read failed after 0 of 4096 at 503316471808: Input/output error
  Couldn't find device with uuid K7Xklz-OruW-4kDk-1GbN-tEPz-xASw-ZoWCqB.
    /dev/dm-16: read failed after 0 of 4096 at 16710041600: Input/output error
    /dev/dm-16: read failed after 0 of 4096 at 16710098944: Input/output error
    /dev/dm-16: read failed after 0 of 4096 at 0: Input/output error
    /dev/dm-16: read failed after 0 of 4096 at 4096: Input/output error
    /dev/dm-16: read failed after 0 of 4096 at 0: Input/output error
    /dev/dm-17: read failed after 0 of 4096 at 52613283840: Input/output error
    /dev/dm-17: read failed after 0 of 4096 at 52613341184: Input/output error
    /dev/dm-17: read failed after 0 of 4096 at 0: Input/output error
    /dev/dm-17: read failed after 0 of 4096 at 4096: Input/output error
    /dev/dm-17: read failed after 0 of 4096 at 0: Input/output error
    /dev/dm-18: read failed after 0 of 4096 at 42949607424: Input/output error
    /dev/dm-18: read failed after 0 of 4096 at 42949664768: Input/output error
    /dev/dm-18: read failed after 0 of 4096 at 0: Input/output error
    /dev/dm-18: read failed after 0 of 4096 at 4096: Input/output error
    /dev/dm-18: read failed after 0 of 4096 at 0: Input/output error
    /dev/dm-21: read failed after 0 of 4096 at 503316414464: Input/output error
    /dev/dm-21: read failed after 0 of 4096 at 503316471808: Input/output error
    Couldn't find device with uuid K7Xklz-OruW-4kDk-1GbN-tEPz-xASw-ZoWCqB.
    There are 1 physical volumes missing.
  LV                    VG         #Seg Attr   LSize   Maj Min KMaj KMin Origin Snap%  Move Copy%  Log Convert LV UUID                                Devices
  LVhome                VolGroup00    1 mwi-a- 468.75g  -1  -1 254  21                       59.63             JXflDp-bh8q-TLQ7-wUr9-2vDV-E151-RrwuvW LVhome_mimage_0(0),LVhome_mimage_1(0)
  [LVhome_mimage_0]     VolGroup00    6 Iwi-ao 468.75g  -1  -1 254  19                                         Ydd8BX-nH0B-aObV-n4rR-4D3c-HPw5-T05FQC /dev/sdc1(19076)
  [LVhome_mimage_0]     VolGroup00    6 Iwi-ao 468.75g  -1  -1 254  19                                         Ydd8BX-nH0B-aObV-n4rR-4D3c-HPw5-T05FQC unknown device(25052)
  [LVhome_mimage_0]     VolGroup00    6 Iwi-ao 468.75g  -1  -1 254  19                                         Ydd8BX-nH0B-aObV-n4rR-4D3c-HPw5-T05FQC unknown device(28240)
  [LVhome_mimage_0]     VolGroup00    6 Iwi-ao 468.75g  -1  -1 254  19                                         Ydd8BX-nH0B-aObV-n4rR-4D3c-HPw5-T05FQC unknown device(14400)
  [LVhome_mimage_0]     VolGroup00    6 Iwi-ao 468.75g  -1  -1 254  19                                         Ydd8BX-nH0B-aObV-n4rR-4D3c-HPw5-T05FQC /dev/sda2(26906)
  [LVhome_mimage_0]     VolGroup00    6 Iwi-ao 468.75g  -1  -1 254  19                                         Ydd8BX-nH0B-aObV-n4rR-4D3c-HPw5-T05FQC /dev/sda2(27556)
  [LVhome_mimage_1]     VolGroup00    4 Iwi-ao 468.75g  -1  -1 254  20                                         3f4tfM-lOWt-ofA0-ruln-DGbI-cs4l-sAaOqM /dev/sda2(11090)
  [LVhome_mimage_1]     VolGroup00    4 Iwi-ao 468.75g  -1  -1 254  20                                         3f4tfM-lOWt-ofA0-ruln-DGbI-cs4l-sAaOqM /dev/sda2(24058)
  [LVhome_mimage_1]     VolGroup00    4 Iwi-ao 468.75g  -1  -1 254  20                                         3f4tfM-lOWt-ofA0-ruln-DGbI-cs4l-sAaOqM /dev/sdc1(25780)
  [LVhome_mimage_1]     VolGroup00    4 Iwi-ao 468.75g  -1  -1 254  20                                         3f4tfM-lOWt-ofA0-ruln-DGbI-cs4l-sAaOqM /dev/sdc1(10528)
I'm unable to recover from this, and ALL of my critical data is in this LV. I thought using a Mirror would prevent a drive failure from causing me to lose data... but it seems no configuration, even ones intended to allow you to proceed with a missing mirrored drive, behaves in an expected way.

I've tried splitting the mirror, repairing, converting to linear... nothing works. Everything tells me that it won't make any change to the VG with missing PVs. The copy% you see above is when I run lvchange -ay --partial VolGroup00/LVhome - this seems to be the only way to get lvm to do anything with this LV. When I mount it, I get a lot of garbage in the ls. Really? Two copies, and I've got garbage?

Has anyone else run into this? I suspect part of the problem being that mirror 1 lies on two separate PVs (as well as mirror 0). I should have access to all of my data. Not even fsck will run without an error that scares the bejeezus out of me.

As a side note, I had other LVs that were mirrored, with one leg on the failed PV, and I had no problem recovering them. But, they had each leg entirely on a single PV. The LV I have a problem with has been resized a few times to add space, since this is the /home mount. Apparently, the resize operation didn't keep the legs split across the PVs like it should. Good thing I didn't lose the PV that mirror 1 sits on, since mirror 0 also straddles it.

Code:
# e2fsck -p -f /dev/mapper/VolGroup00-LVhome
Error reading block 64211540 (Attempt to read block from filesystem resulted in short read) while reading indirect blocks of inode 426151.

/dev/mapper/VolGroup00-LVhome: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
        (i.e., without -a or -p options)
Code:
# lvconvert --splitmirrors 1 --name homecopy VolGroup00/LVhome
  /dev/dm-16: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-17: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-18: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-16: read failed after 0 of 4096 at 16710041600: Input/output error
  /dev/dm-16: read failed after 0 of 4096 at 16710098944: Input/output error
  /dev/dm-16: read failed after 0 of 4096 at 4096: Input/output error
  /dev/dm-17: read failed after 0 of 4096 at 52613283840: Input/output error
  /dev/dm-17: read failed after 0 of 4096 at 52613341184: Input/output error
  /dev/dm-17: read failed after 0 of 4096 at 4096: Input/output error
  /dev/dm-18: read failed after 0 of 4096 at 42949607424: Input/output error
  /dev/dm-18: read failed after 0 of 4096 at 42949664768: Input/output error
  /dev/dm-18: read failed after 0 of 4096 at 4096: Input/output error
  /dev/dm-21: read failed after 0 of 4096 at 503316414464: Input/output error
  /dev/dm-21: read failed after 0 of 4096 at 503316471808: Input/output error
  Couldn't find device with uuid K7Xklz-OruW-4kDk-1GbN-tEPz-xASw-ZoWCqB.
  Cannot change VG VolGroup00 while PVs are missing.
  Consider vgreduce --removemissing.
If I could just mount the intact mirror by itself, I'd be in good shape. I can't even add another PV to the VG in this situation to give me more space.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: How To Set Up Software RAID1 On A Running LVM System (Debian Squeeze) LXer Syndicated Linux News 0 06-15-2011 07:30 AM
RAID 10 failed - and my installation uses a lvm on top - how can i recover from this? nicolasdiogo Linux - General 5 05-20-2011 12:22 PM
Splitting a mirror with LVM jcaplette Linux - Server 1 03-09-2010 03:14 AM
Replace failed LVM mirror leg eldo Linux - Server 1 11-20-2009 02:19 AM
LVM - unable to mirror LV membit Linux - Server 1 06-20-2009 02:42 AM


All times are GMT -5. The time now is 09:34 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration