LVM mysteriously dissapeared
 

Hello,

Been using LVM for a while and absolutely love it, but today ran into an
"interesting" problem.

All of a sudden all logical volumes on one of my boxes have gone
"missing".

bfs:~# lvdisplay
No volume groups found
bfs:~# vgdisplay
No volume groups found
bfs:~# pvdisplay

The box has four disks. Two small ones are RAID1 (/dev/md0) and
contain the / fs. Two big disks are also RAID1
(/dev/md1) and contained a single partition which was allocated as a PV
to LVM. RAID volumes are healthy according to /cat/proc/mdstat and
mdadm, however fdisk claims there isn't a valid partition table
on /dev/md1 or /dev/md0. Is it correct to say that might be the root
cause?

In /etc/lvm/backup and /etc/lvm/archive I have info about my LVM setup,
with the latest being from two months ago which was the last time I
created an LV (may I just say that keeping this info is just genius on
part LVM creators?).

Any suggestions on how to move forward with this investigation and hopefully repair? Where on disk does LVM keep it's information?

thanks much in advance,
Alain

Try running "vgchange -ay" to manually activate the volumes. This command is normally run at boot, but maybe somehow it failed on your last boot???

[edit]p.s. - you may need to run "pvscan" before that vgchange command - I can't remember if that is required or not.[/edit]

It is genius so long as you don't happen to have /etc/lvm/backup on an LVM volume itself! So copy that data off-computer, to some external backup somewhere.

haertig,

Thank you for your reply!

Unfortunately I get:

bfs:~# vgchange -ay
No volume groups found
bfs:~# pvscan
No matching physical volumes found

I know that LVM worked fine last time the box booted because it's a Xen host and the VM filesystems live on logical volumes. For instance if I rebooted now or restarted the virtual machines, they'd fail to start. This is actually how I learned of LVM issues on this box; one of the VMs was acting weird, I rebooted it and it wouldn't come up. The other VMs are currently up and running, which gives me a change to make sure all my backups are up to date.

This box has been up for 133 days, which is when I think there was a kernel update I installed.

I scoured the logs for any errors and there's nothing. This is all very strange...

You're quite right. This is exactly why I've always resisted putting the root filesystem on LVM. LVM adds cool features, cool features add complexity, complexity leads to more stuff that can break.

Are the disks that contain the volumes even seen by the system? i.e., Does "fdisk -l" show you their existence?

You could also try booting a recent Knoppix or other LiveCD and trying to access the volumes. You will probably need to run the "pvscan" and "vgchange -ay" manually if you boot from a LiveCD. Booting from a LiveCD might tell you if you're dealing with a (1) a problem in your volumes, or (2) a problem in the installed software on your system that accesses the volumes.

Yeah, fdisk sees them. I'm still working on setting up a new Xen host to move the functionality over, once done, I'll try rebooting and perhaps booting to a Live CD to see what it sees.

This just underscores to me the importance of having some sort of application level redundancy and failover. In this case I don't have it as it's my home system. But because restoring from backup is such a PITA and I invariably end up loosing some data, I should have redundancy at home too (double the NFS/Samba, double the DNS, double the SMTP/IMAP and double the LAMP). Good times. :-\

RAID devices are healthy

You have lots of disks, therefore lots of PV's. I find it hard to believe that pvscan can't find ANY of them. You did check pvscan, right? Always start at the lowest level, the disk/partition - which you already verified, then move on to the PV level, before continuing on to VG's and LV's. Even if the volume group(s)/logical volume(s) is/are corrupted you'd still see the PV's. The odds of hosing up all of your PV's simply defies my understanding, so I'd be looking for something else that they all have in common as the cause. The only thing I can think of at the moment that they all share (at the PV level) is the pvscan command itself. Even with no LV's or VG's (or corrupted ones), the PV's should be visible. I don't know what to expect, but were it my system the first thing I'd try is a fresh copy of pvscan and it's required libraries - i.e., boot with a LiveCD. I'm not sure what else you could investigate right now.

I've been assuming that you are already familiar with pvscan, vgscan, etc. normal output. But maybe you aren't. In case not, here's some sample (normal) output from one of my home systems. You can see I multi-boot several different distros and have a somewhat-complex setup! And I use partitions, not entire disks, for my PV's.

Yep, pvscan, vgscan and any other LVM command makes it seem like LVM was never even setup on this box. If I didn't have the evidence to the contrary in /etc/lvm/, I would be considering a very strong possibility that I've finally and completely lost my mind.

I also use partitions instead of disks for LVM as that is strongly recommended by http://tldp.org/HOWTO/LVM-HOWTO/initdisks.html

That makes a lot of sense. I'll try a LiveCD as soon as I'm finished transferring important stuff to another box and feel adventurous enough to start digging further. ;)

If you're saying you are using partitions rather than entire disks for your PV's, then fdisk shouldn't be giving you that "does not contain a valid partition table" warning. When I saw that, I just assumed you were using entire disks for your PV's, as you will get this warning in that scenario.

So it now appears that your partition table(s) are wiped out? All of them, on all of the disks? That's just as mystifying to me as if entire disk PV's had been wiped out instead. But to be honest, a partition table wipe-out will probably be easier to deal with than and LVM wipeout. I'd say your chances of recovering a partition table are better.

Did you backup your partition table(s) before this disaster befell you? You might be able to use dd to extract your partition table to a simple file. It's only 16 bytes long (I think?) Very small, anyway. You could manually look analyze it byte-by-byte and see if it looks anywhere near sane (and repairable) or if it's total garbage. I used to know the exact makeup of a partition table off the top of my head, but I've long since forgotten the details. A Google search is your friend on this.

Yep, that's what I was thinking...
It's worth noting that the partition information is missing from /dev/md0 also, where my root fs lives (whichs is why I'm in no hurry to reboot) ;). The partition setup on /dev/md1 was simple; just one big partition which was used as a PV for LVM.

I usually save my partition and RAID setup info to help me out when disks fail, so I have to go dig that up.

haertig, I appreciate your time thinking about this problem. While stuff breaking is an inconvenience, this is when I find myself learning the most, so it's hard to complain.. :study:

Cheers,
Alain

Do you recall where on disk partition info is kept and where LVM keeps its stuff? What's a good utility to examine parts of disk that contain partition and LVM info?