LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Hardware
User Name
Password
Linux - Hardware This forum is for Hardware issues.
Having trouble installing a piece of hardware? Want to know if that peripheral is compatible with Linux?

Notices

Reply
 
Search this Thread
Old 03-19-2009, 09:27 AM   #1
x1um1n
LQ Newbie
 
Registered: Mar 2009
Location: Wirral, UK
Distribution: Ubuntu, Gentoo
Posts: 9

Rep: Reputation: 0
LVM Nightmare, how do I remove FUBAR hdd's?


Hi Guys,
A couple of weeks ago I stuck a new drive in my lvm array, it worked fine for about 3 days then started making that horrible click-scrape, click scrape noise you get when the heads die..

I've been struggling since then to remove the bloody thing from my array, but whenever I boot it up it either doesn't find the drive, or it only finds it briefly.

This means I can't just lvreduce it.

I'm about at my wits end..

If I remove the hdd, is there any way I can recover the rest of the array?

atm pvscan (or any other lvm command) just tells me it can't find all the pv's in my vg and so can't do anything..


I've done quite a lot of googling and haven't been able to find anything other than hints that I might be able to forcibly remove the pv's. But I can't find anything that tells me HOW?

Please help!

--

"Computer, if you don't open that hatch right now I'm going to go to your major databanks with a large axe and give you a reprogramming you'll never forget!"

Zaphod Beeblebrox
 
Old 03-19-2009, 01:23 PM   #2
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,350

Rep: Reputation: 1041Reputation: 1041Reputation: 1041Reputation: 1041Reputation: 1041Reputation: 1041Reputation: 1041Reputation: 1041
I don't use LVM (for several reasons, you may have just found another one), but the manpage for vgcfgrestore indicates you might be able to work around this if you are replacing the disk. Got a(nother) spare ?.
This bit is a bit of a worry ...
Quote:
If a PV in the VG is lost and you wish to substitute another of the same size ...
It doesn't offer any option if you want to use a different size - I guess pvcreate forcing the size might work presuming you use a larger disk.

Good luck.

<Edit:>
LVM is *not* for data integrity - you still need backups. You'll need to restore the LV(s) once the VG is re-built.
</Edit:>

Last edited by syg00; 03-19-2009 at 01:27 PM.
 
Old 03-19-2009, 01:49 PM   #3
lazlow
Senior Member
 
Registered: Jan 2006
Posts: 4,362

Rep: Reputation: 171Reputation: 171
Not able to help you with this issue, but when you go to rebuild I can tell you how to do it without using LVMs and get almost (see below) all the same functionality. The only feature that cannot be handled in other(simpler, safer) ways is contiguous space (one large file across multiple drives). You have just found one of the MANY PITA problems with LVMs, which for most users offer no extra functionality.
 
Old 03-19-2009, 02:00 PM   #4
x1um1n
LQ Newbie
 
Registered: Mar 2009
Location: Wirral, UK
Distribution: Ubuntu, Gentoo
Posts: 9

Original Poster
Rep: Reputation: 0
Unhappy

Thanks Guys,
unfortunately the disk in question is a 1.5Tb and from what I've been reading these things are flaky as hell (as I'm now discovering first hand) so I was planning on replacing it with a 1Tb.

Also, the reason I use LVM is for it's ability to spread a partition (currently 5.6Tb) across multiple disks.

I'm beginning to wish I'd just broken the tree down into a less elegant but more stable solution:

/blah/1 --> sda
/blah/2 --> sdb
/blah/3 --> sdc

Unless you can suggest a better option?

In the meantime I'll have a look into vgcfgrestore..
 
Old 03-19-2009, 02:31 PM   #5
lazlow
Senior Member
 
Registered: Jan 2006
Posts: 4,362

Rep: Reputation: 171Reputation: 171
Yep, keep each drive independent. That way any drive fails you only have to deal with just that. You can setup your mount points on the master drive (sda usually) in such a manner that the casual user cannot tell they are actually using a different drive. I have /home on a separate drive (mount point on master) and I have a /data subdirectory in each users home that is in reality on a third drive. Once you understand mount points and using fstab it is a breeze to set this up or rearrange it.
 
Old 03-19-2009, 02:32 PM   #6
x1um1n
LQ Newbie
 
Registered: Mar 2009
Location: Wirral, UK
Distribution: Ubuntu, Gentoo
Posts: 9

Original Poster
Rep: Reputation: 0
Angry

Ok, vgcfgrestore can restore the array to the state it was in in a backup file.

Unfortunately the backup doesn't go far back enough..

It looks like it's not 1 disk that's died, but 2.

Code:
root@omnius:~# pvscan 
File descriptor 5 left open
  Couldn't find device with uuid 'XeBtci-t5fV-pvlz-Bwu9-CXqG-niq7-KjPgTV'.
  Couldn't find device with uuid 'CJEQAz-CenQ-Kn7q-4nZ5-X0oE-2s3F-V4AEcw'.
  Couldn't find device with uuid 'XeBtci-t5fV-pvlz-Bwu9-CXqG-niq7-KjPgTV'.
  Couldn't find device with uuid 'CJEQAz-CenQ-Kn7q-4nZ5-X0oE-2s3F-V4AEcw'.
  Couldn't find device with uuid 'XeBtci-t5fV-pvlz-Bwu9-CXqG-niq7-KjPgTV'.
  Couldn't find device with uuid 'CJEQAz-CenQ-Kn7q-4nZ5-X0oE-2s3F-V4AEcw'.
  Couldn't find device with uuid 'XeBtci-t5fV-pvlz-Bwu9-CXqG-niq7-KjPgTV'.
  Couldn't find device with uuid 'CJEQAz-CenQ-Kn7q-4nZ5-X0oE-2s3F-V4AEcw'.
  Couldn't find device with uuid 'XeBtci-t5fV-pvlz-Bwu9-CXqG-niq7-KjPgTV'.
  Couldn't find device with uuid 'CJEQAz-CenQ-Kn7q-4nZ5-X0oE-2s3F-V4AEcw'.
  Couldn't find device with uuid 'XeBtci-t5fV-pvlz-Bwu9-CXqG-niq7-KjPgTV'.
  Couldn't find device with uuid 'CJEQAz-CenQ-Kn7q-4nZ5-X0oE-2s3F-V4AEcw'.
  Couldn't find device with uuid 'XeBtci-t5fV-pvlz-Bwu9-CXqG-niq7-KjPgTV'.
  Couldn't find device with uuid 'CJEQAz-CenQ-Kn7q-4nZ5-X0oE-2s3F-V4AEcw'.
  Couldn't find device with uuid 'XeBtci-t5fV-pvlz-Bwu9-CXqG-niq7-KjPgTV'.
  Couldn't find device with uuid 'CJEQAz-CenQ-Kn7q-4nZ5-X0oE-2s3F-V4AEcw'.
  Couldn't find device with uuid 'XeBtci-t5fV-pvlz-Bwu9-CXqG-niq7-KjPgTV'.
  Couldn't find device with uuid 'CJEQAz-CenQ-Kn7q-4nZ5-X0oE-2s3F-V4AEcw'.
  Couldn't find device with uuid 'XeBtci-t5fV-pvlz-Bwu9-CXqG-niq7-KjPgTV'.
  Couldn't find device with uuid 'CJEQAz-CenQ-Kn7q-4nZ5-X0oE-2s3F-V4AEcw'.
  PV /dev/sdc1        VG omnius_vg   lvm2 [931.51 GB / 0    free]
  PV /dev/sdb1        VG omnius_vg   lvm2 [931.51 GB / 0    free]
  PV /dev/sda4        VG omnius_vg   lvm2 [670.11 GB / 0    free]
  PV /dev/sdd1        VG omnius_vg   lvm2 [931.51 GB / 0    free]
  PV unknown device   VG omnius_vg   lvm2 [1.36 TB / 0    free]
  PV unknown device   VG omnius_vg   lvm2 [1.36 TB / 972.00 MB free]
  Total: 6 [6.11 TB] / in use: 6 [6.11 TB] / in no VG: 0 [0   ]
The oldest backup lvm's got is just before I added the last disk and if I try to restore from it, I get this:

Code:
root@omnius:~# vgcfgrestore -f /etc/lvm/archive/omnius_vg_00000.vg omnius_vgFile descriptor 5 left open
  Couldn't find device with uuid 'XeBtci-t5fV-pvlz-Bwu9-CXqG-niq7-KjPgTV'.
  Couldn't find all physical volumes for volume group omnius_vg.
  Restore failed.
I'm going to try again in a few hours, the kernel seems to pick the disks up intermittently, maybe if I leave the newest disk out it'll be able to see the slightly older one and I can try and lvreduce/resize2fs that one out of the array :$
 
Old 03-19-2009, 02:38 PM   #7
x1um1n
LQ Newbie
 
Registered: Mar 2009
Location: Wirral, UK
Distribution: Ubuntu, Gentoo
Posts: 9

Original Poster
Rep: Reputation: 0
:lazlow

unfortunately, this one partition is the equivalent of your /data dir. I used to have it broken down into categories but the volume of data's outgrown that solution. Some categories are WAY bigger than 1Tb, others are only 200Gb.

That's why I turned to lvm in the first place.
 
Old 03-19-2009, 03:15 PM   #8
lazlow
Senior Member
 
Registered: Jan 2006
Posts: 4,362

Rep: Reputation: 171Reputation: 171
If you are getting that big skip LVM and go straight to raid. Go ahead and spend the $300 for a true hardware raid card. I know it seems like a ton of money but in the long run it will save you time and hair.
 
Old 03-20-2009, 08:25 PM   #9
x1um1n
LQ Newbie
 
Registered: Mar 2009
Location: Wirral, UK
Distribution: Ubuntu, Gentoo
Posts: 9

Original Poster
Rep: Reputation: 0
Reet, managed to get it sorted, mostly..

I used vgcfgrestore to revert to how it was before the last drive was put in. Then physically removed that drive from the box.

The other drive which is dying (but not yet dead) I've managed to deal with by lvresize -L-1.4T /dev/vg/lv, now I'm just trying to get resize2fs to shrink the fs to match the partition size (then I can see how much data I've lost )

thank fsck for backups..

unfortunately, while the other drive was being relatively friendly yday, it's being less so today..

Thanks for all your help on this one guys
 
Old 03-20-2009, 09:43 PM   #10
x1um1n
LQ Newbie
 
Registered: Mar 2009
Location: Wirral, UK
Distribution: Ubuntu, Gentoo
Posts: 9

Original Poster
Rep: Reputation: 0
Talking

Reet, resize2fs wasn't happy, it claimed:

"resize2fs: Can't read an block bitmap while trying to resize"

which apart from being almost as grammatically sound as "all your base are belong to us" isn't tremendously helpful

I stuck it in google, (it was the only google suggestion btw, worrying when that happens) and the only help I could find was, pull the data off and start afresh..

fortunately there's only 1.3Tb ish of data which isn't backed up, and now I've got the fubar drives out I can swap em for some slightly more stable 1Tb drives, this shouldn't be a problem.

I can still mount the partition, but I'm guessing it wouldn't be too happy if I tried to put anything in the final 3Tb of it

---

Now, does anyone know who to email to get this fixed? lvm's been great, 'til now, but a storage array that becomes completely useless when 1 hdd fails is pathetic. This needs sorting out, while ubu doesn't use lvm except under duress, fedora has been using it by default for the last couple years and NixOS (a VERY interesting new distro, check it out if you haven't heard of it) can't really handle anything else.

Most worryingly of all, last time I played with Enterprise *nix (RHEL4) i'm pretty sure it wanted to LVM at install time..

this needs fixing!
 
Old 03-20-2009, 10:05 PM   #11
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,350

Rep: Reputation: 1041Reputation: 1041Reputation: 1041Reputation: 1041Reputation: 1041Reputation: 1041Reputation: 1041Reputation: 1041
Try the mail list mentioned here.
I suspect they'll tell you to add new drives and (after the vgcfgrestore) restore the data.

As for NixOS, I keep meaning to look at it, but I'm really more interested in Nix itself.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Can I remove an LVM easily? Exospaciac Linux - Software 1 08-10-2008 06:27 PM
Correct Way To Remove a Drive from LVM kromberg Linux - Software 1 12-10-2007 09:27 PM
LVM Nightmare! Have I lost my data? MythN00b Linux - Hardware 14 09-20-2007 09:42 PM
LVM - can't remove vg zdenisl Linux - Software 1 08-16-2007 08:36 AM
Advisability of using LVM on backup HDD's? haertig Linux - Software 0 12-28-2005 08:43 PM


All times are GMT -5. The time now is 04:21 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration