Linux - SoftwareThis forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Distribution: debian on servers, ubuntu on desktops/laptops
Posts: 45
Rep:
LVM mysteriously dissapeared
Hello,
Been using LVM for a while and absolutely love it, but today ran into an
"interesting" problem.
All of a sudden all logical volumes on one of my boxes have gone
"missing".
bfs:~# lvdisplay
No volume groups found
bfs:~# vgdisplay
No volume groups found
bfs:~# pvdisplay
The box has four disks. Two small ones are RAID1 (/dev/md0) and
contain the / fs. Two big disks are also RAID1
(/dev/md1) and contained a single partition which was allocated as a PV
to LVM. RAID volumes are healthy according to /cat/proc/mdstat and
mdadm, however fdisk claims there isn't a valid partition table
on /dev/md1 or /dev/md0. Is it correct to say that might be the root
cause?
In /etc/lvm/backup and /etc/lvm/archive I have info about my LVM setup,
with the latest being from two months ago which was the last time I
created an LV (may I just say that keeping this info is just genius on
part LVM creators?).
Any suggestions on how to move forward with this investigation and hopefully repair? Where on disk does LVM keep it's information?
In /etc/lvm/backup and /etc/lvm/archive I have info about my LVM setup, with the latest being from two months ago which was the last time I created an LV (may I just say that keeping this info is just genius on part LVM creators?).
It is genius so long as you don't happen to have /etc/lvm/backup on an LVM volume itself! So copy that data off-computer, to some external backup somewhere.
Distribution: debian on servers, ubuntu on desktops/laptops
Posts: 45
Original Poster
Rep:
haertig,
Thank you for your reply!
Quote:
Originally Posted by haertig
Try running "vgchange -ay" to manually activate the volumes.
Unfortunately I get:
bfs:~# vgchange -ay
No volume groups found
bfs:~# pvscan
No matching physical volumes found
Quote:
Originally Posted by haertig
This command is normally run at boot, but maybe somehow it failed on your last boot???
I know that LVM worked fine last time the box booted because it's a Xen host and the VM filesystems live on logical volumes. For instance if I rebooted now or restarted the virtual machines, they'd fail to start. This is actually how I learned of LVM issues on this box; one of the VMs was acting weird, I rebooted it and it wouldn't come up. The other VMs are currently up and running, which gives me a change to make sure all my backups are up to date.
This box has been up for 133 days, which is when I think there was a kernel update I installed.
I scoured the logs for any errors and there's nothing. This is all very strange...
Distribution: debian on servers, ubuntu on desktops/laptops
Posts: 45
Original Poster
Rep:
Quote:
Originally Posted by haertig
It is genius so long as you don't happen to have /etc/lvm/backup on an LVM volume itself! So copy that data off-computer, to some external backup somewhere.
You're quite right. This is exactly why I've always resisted putting the root filesystem on LVM. LVM adds cool features, cool features add complexity, complexity leads to more stuff that can break.
Are the disks that contain the volumes even seen by the system? i.e., Does "fdisk -l" show you their existence?
You could also try booting a recent Knoppix or other LiveCD and trying to access the volumes. You will probably need to run the "pvscan" and "vgchange -ay" manually if you boot from a LiveCD. Booting from a LiveCD might tell you if you're dealing with a (1) a problem in your volumes, or (2) a problem in the installed software on your system that accesses the volumes.
Distribution: debian on servers, ubuntu on desktops/laptops
Posts: 45
Original Poster
Rep:
Yeah, fdisk sees them. I'm still working on setting up a new Xen host to move the functionality over, once done, I'll try rebooting and perhaps booting to a Live CD to see what it sees.
This just underscores to me the importance of having some sort of application level redundancy and failover. In this case I don't have it as it's my home system. But because restoring from backup is such a PITA and I invariably end up loosing some data, I should have redundancy at home too (double the NFS/Samba, double the DNS, double the SMTP/IMAP and double the LAMP). Good times. :-\
Quote:
bfs:~# fdisk -l
Disk /dev/sda: 160.0 GB, 160000000000 bytes
255 heads, 63 sectors/track, 19452 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sda doesn't contain a valid partition table
Disk /dev/sdb: 160.0 GB, 160000000000 bytes
255 heads, 63 sectors/track, 19452 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdb doesn't contain a valid partition table
Disk /dev/sdc: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdc doesn't contain a valid partition table
Disk /dev/sdd: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdd doesn't contain a valid partition table
Disk /dev/md0: 157.9 GB, 157999300608 bytes
2 heads, 4 sectors/track, 38574048 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk /dev/md0 doesn't contain a valid partition table
Disk /dev/md1: 500.1 GB, 500105150464 bytes
2 heads, 4 sectors/track, 122095984 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk /dev/md1 doesn't contain a valid partition table
Disk /dev/dm-0: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-0 doesn't contain a valid partition table
Disk /dev/dm-1: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-1 doesn't contain a valid partition table
Disk /dev/dm-2: 322.1 GB, 322122547200 bytes
255 heads, 63 sectors/track, 39162 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-2 doesn't contain a valid partition table
Disk /dev/dm-3: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-3 doesn't contain a valid partition table
Disk /dev/dm-4: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-4 doesn't contain a valid partition table
Disk /dev/dm-5: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-5 doesn't contain a valid partition table
Disk /dev/dm-6: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-6 doesn't contain a valid partition table
Disk /dev/dm-7: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-7 doesn't contain a valid partition table
Disk /dev/dm-8: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-8 doesn't contain a valid partition table
Disk /dev/dm-9: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-9 doesn't contain a valid partition table
Disk /dev/dm-10: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-10 doesn't contain a valid partition table
Disk /dev/dm-11: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-11 doesn't contain a valid partition table
Disk /dev/dm-12: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-12 doesn't contain a valid partition table
Disk /dev/dm-13: 7516 MB, 7516192768 bytes
255 heads, 63 sectors/track, 913 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-13 doesn't contain a valid partition table
You have lots of disks, therefore lots of PV's. I find it hard to believe that pvscan can't find ANY of them. You did check pvscan, right? Always start at the lowest level, the disk/partition - which you already verified, then move on to the PV level, before continuing on to VG's and LV's. Even if the volume group(s)/logical volume(s) is/are corrupted you'd still see the PV's. The odds of hosing up all of your PV's simply defies my understanding, so I'd be looking for something else that they all have in common as the cause. The only thing I can think of at the moment that they all share (at the PV level) is the pvscan command itself. Even with no LV's or VG's (or corrupted ones), the PV's should be visible. I don't know what to expect, but were it my system the first thing I'd try is a fresh copy of pvscan and it's required libraries - i.e., boot with a LiveCD. I'm not sure what else you could investigate right now.
I've been assuming that you are already familiar with pvscan, vgscan, etc. normal output. But maybe you aren't. In case not, here's some sample (normal) output from one of my home systems. You can see I multi-boot several different distros and have a somewhat-complex setup! And I use partitions, not entire disks, for my PV's.
Code:
root@FamilyRoom:~# pvscan
PV /dev/sdb6 VG vg_fileserver lvm2 [9.31 GB / 0 free]
PV /dev/sdb7 VG vg_fileserver lvm2 [9.31 GB / 0 free]
PV /dev/sdb8 VG vg_fileserver lvm2 [9.31 GB / 0 free]
PV /dev/sdb9 VG vg_fileserver lvm2 [9.31 GB / 4.26 GB free]
PV /dev/sdb5 VG vg_backup lvm2 [9.31 GB / 0 free]
PV /dev/sda9 VG vg_fedora_9 lvm2 [9.54 GB / 3.16 GB free]
PV /dev/sda8 VG vg_opensuse_11.0 lvm2 [9.54 GB / 3.61 GB free]
PV /dev/sda7 VG vg_debian_4.0r4a lvm2 [9.54 GB / 2.62 GB free]
PV /dev/sda6 VG vg_ubuntu_8.04.1 lvm2 [9.54 GB / 2.88 GB free]
PV /dev/sda5 VG vg_slackware_12.1 lvm2 [9.54 GB / 4.16 GB free]
PV /dev/sda10 lvm2 [9.54 GB]
Total: 11 [103.81 GB] / in use: 10 [94.27 GB] / in no VG: 1 [9.54 GB]
root@FamilyRoom:~# pvs
PV VG Fmt Attr PSize PFree
/dev/sda10 lvm2 -- 9.54G 9.54G
/dev/sda5 vg_slackware_12.1 lvm2 a- 9.54G 4.16G
/dev/sda6 vg_ubuntu_8.04.1 lvm2 a- 9.54G 2.88G
/dev/sda7 vg_debian_4.0r4a lvm2 a- 9.54G 2.62G
/dev/sda8 vg_opensuse_11.0 lvm2 a- 9.54G 3.61G
/dev/sda9 vg_fedora_9 lvm2 a- 9.54G 3.16G
/dev/sdb5 vg_backup lvm2 a- 9.31G 0
/dev/sdb6 vg_fileserver lvm2 a- 9.31G 0
/dev/sdb7 vg_fileserver lvm2 a- 9.31G 0
/dev/sdb8 vg_fileserver lvm2 a- 9.31G 0
/dev/sdb9 vg_fileserver lvm2 a- 9.31G 4.26G
root@FamilyRoom:~# vgscan
Reading all physical volumes. This may take a while...
Found volume group "vg_fileserver" using metadata type lvm2
Found volume group "vg_backup" using metadata type lvm2
Found volume group "vg_fedora_9" using metadata type lvm2
Found volume group "vg_opensuse_11.0" using metadata type lvm2
Found volume group "vg_debian_4.0r4a" using metadata type lvm2
Found volume group "vg_ubuntu_8.04.1" using metadata type lvm2
Found volume group "vg_slackware_12.1" using metadata type lvm2
root@FamilyRoom:~# vgs
VG #PV #LV #SN Attr VSize VFree
vg_backup 1 1 0 wz--n- 9.31G 0
vg_debian_4.0r4a 1 7 0 wz--n- 9.54G 2.62G
vg_fedora_9 1 7 0 wz--n- 9.54G 3.16G
vg_fileserver 4 3 0 wz--n- 37.25G 4.26G
vg_opensuse_11.0 1 7 0 wz--n- 9.54G 3.61G
vg_slackware_12.1 1 7 0 wz--n- 9.54G 4.16G
vg_ubuntu_8.04.1 1 7 0 wz--n- 9.54G 2.88G
root@FamilyRoom:~# lvscan
ACTIVE '/dev/vg_fileserver/lv_music' [19.99 GB] inherit
ACTIVE '/dev/vg_fileserver/lv_photography' [12.00 GB] inherit
ACTIVE '/dev/vg_fileserver/lv_data' [1.00 GB] inherit
ACTIVE '/dev/vg_backup/lv_backup' [9.31 GB] inherit
ACTIVE '/dev/vg_fedora_9/lv_home' [104.00 MB] inherit
ACTIVE '/dev/vg_fedora_9/lv_tmp' [52.00 MB] inherit
ACTIVE '/dev/vg_fedora_9/lv_opt' [52.00 MB] inherit
ACTIVE '/dev/vg_fedora_9/lv_var' [404.00 MB] inherit
ACTIVE '/dev/vg_fedora_9/lv_swap' [500.00 MB] inherit
ACTIVE '/dev/vg_fedora_9/lv_root' [300.00 MB] inherit
ACTIVE '/dev/vg_fedora_9/lv_usr' [5.00 GB] inherit
ACTIVE '/dev/vg_opensuse_11.0/lv_swap' [52.00 MB] inherit
ACTIVE '/dev/vg_opensuse_11.0/lv_tmp' [52.00 MB] inherit
ACTIVE '/dev/vg_opensuse_11.0/lv_home' [52.00 MB] inherit
ACTIVE '/dev/vg_opensuse_11.0/lv_opt' [252.00 MB] inherit
ACTIVE '/dev/vg_opensuse_11.0/lv_var' [152.00 MB] inherit
ACTIVE '/dev/vg_opensuse_11.0/lv_root' [400.00 MB] inherit
ACTIVE '/dev/vg_opensuse_11.0/lv_usr' [5.00 GB] inherit
ACTIVE '/dev/vg_debian_4.0r4a/lv_swap' [500.00 MB] inherit
ACTIVE '/dev/vg_debian_4.0r4a/lv_root' [300.00 MB] inherit
ACTIVE '/dev/vg_debian_4.0r4a/lv_usr' [5.00 GB] inherit
ACTIVE '/dev/vg_debian_4.0r4a/lv_var' [1004.00 MB] inherit
ACTIVE '/dev/vg_debian_4.0r4a/lv_tmp' [52.00 MB] inherit
ACTIVE '/dev/vg_debian_4.0r4a/lv_opt' [52.00 MB] inherit
ACTIVE '/dev/vg_debian_4.0r4a/lv_home' [52.00 MB] inherit
ACTIVE '/dev/vg_ubuntu_8.04.1/lv_swap' [500.00 MB] inherit
ACTIVE '/dev/vg_ubuntu_8.04.1/lv_root' [300.00 MB] inherit
ACTIVE '/dev/vg_ubuntu_8.04.1/lv_usr' [5.00 GB] inherit
ACTIVE '/dev/vg_ubuntu_8.04.1/lv_var' [752.00 MB] inherit
ACTIVE '/dev/vg_ubuntu_8.04.1/lv_tmp' [52.00 MB] inherit
ACTIVE '/dev/vg_ubuntu_8.04.1/lv_opt' [52.00 MB] inherit
ACTIVE '/dev/vg_ubuntu_8.04.1/lv_home' [52.00 MB] inherit
ACTIVE '/dev/vg_slackware_12.1/lv_swap' [500.00 MB] inherit
ACTIVE '/dev/vg_slackware_12.1/lv_home' [52.00 MB] inherit
ACTIVE '/dev/vg_slackware_12.1/lv_opt' [52.00 MB] inherit
ACTIVE '/dev/vg_slackware_12.1/lv_tmp' [52.00 MB] inherit
ACTIVE '/dev/vg_slackware_12.1/lv_var' [52.00 MB] inherit
ACTIVE '/dev/vg_slackware_12.1/lv_usr' [4.50 GB] inherit
ACTIVE '/dev/vg_slackware_12.1/lv_root' [200.00 MB] inherit
root@FamilyRoom:~# lvs
LV VG Attr LSize Origin Snap% Move Log Copy%
lv_backup vg_backup -wi-a- 9.31G
lv_home vg_debian_4.0r4a -wi-a- 52.00M
lv_opt vg_debian_4.0r4a -wi-a- 52.00M
lv_root vg_debian_4.0r4a -wi-a- 300.00M
lv_swap vg_debian_4.0r4a -wi-a- 500.00M
lv_tmp vg_debian_4.0r4a -wi-a- 52.00M
lv_usr vg_debian_4.0r4a -wi-a- 5.00G
lv_var vg_debian_4.0r4a -wi-a- 1004.00M
lv_home vg_fedora_9 -wi-a- 104.00M
lv_opt vg_fedora_9 -wi-a- 52.00M
lv_root vg_fedora_9 -wi-a- 300.00M
lv_swap vg_fedora_9 -wi-a- 500.00M
lv_tmp vg_fedora_9 -wi-a- 52.00M
lv_usr vg_fedora_9 -wi-a- 5.00G
lv_var vg_fedora_9 -wi-a- 404.00M
lv_data vg_fileserver -wi-a- 1.00G
lv_music vg_fileserver -wi-a- 19.99G
lv_photography vg_fileserver -wi-a- 12.00G
lv_home vg_opensuse_11.0 -wi-a- 52.00M
lv_opt vg_opensuse_11.0 -wi-a- 252.00M
lv_root vg_opensuse_11.0 -wi-a- 400.00M
lv_swap vg_opensuse_11.0 -wi-a- 52.00M
lv_tmp vg_opensuse_11.0 -wi-a- 52.00M
lv_usr vg_opensuse_11.0 -wi-a- 5.00G
lv_var vg_opensuse_11.0 -wi-a- 152.00M
lv_home vg_slackware_12.1 -wi-a- 52.00M
lv_opt vg_slackware_12.1 -wi-a- 52.00M
lv_root vg_slackware_12.1 -wi-a- 200.00M
lv_swap vg_slackware_12.1 -wi-a- 500.00M
lv_tmp vg_slackware_12.1 -wi-a- 52.00M
lv_usr vg_slackware_12.1 -wi-a- 4.50G
lv_var vg_slackware_12.1 -wi-a- 52.00M
lv_home vg_ubuntu_8.04.1 -wi-ao 52.00M
lv_opt vg_ubuntu_8.04.1 -wi-ao 52.00M
lv_root vg_ubuntu_8.04.1 -wi-ao 300.00M
lv_swap vg_ubuntu_8.04.1 -wi-ao 500.00M
lv_tmp vg_ubuntu_8.04.1 -wi-ao 52.00M
lv_usr vg_ubuntu_8.04.1 -wi-ao 5.00G
lv_var vg_ubuntu_8.04.1 -wi-ao 752.00M
root@FamilyRoom:~#
Distribution: debian on servers, ubuntu on desktops/laptops
Posts: 45
Original Poster
Rep:
Quote:
Originally Posted by haertig
I've been assuming that you are already familiar with pvscan, vgscan, etc. normal output. But maybe you aren't. In case not, here's some sample (normal) output from one of my home systems. You can see I multi-boot several different distros and have a somewhat-complex setup! And I use partitions, not entire disks, for my PV's.
Yep, pvscan, vgscan and any other LVM command makes it seem like LVM was never even setup on this box. If I didn't have the evidence to the contrary in /etc/lvm/, I would be considering a very strong possibility that I've finally and completely lost my mind.
Distribution: debian on servers, ubuntu on desktops/laptops
Posts: 45
Original Poster
Rep:
Quote:
Originally Posted by haertig
The only thing I can think of at the moment that they all share (at the PV level) is the pvscan command itself.
That makes a lot of sense. I'll try a LiveCD as soon as I'm finished transferring important stuff to another box and feel adventurous enough to start digging further.
If you're saying you are using partitions rather than entire disks for your PV's, then fdisk shouldn't be giving you that "does not contain a valid partition table" warning. When I saw that, I just assumed you were using entire disks for your PV's, as you will get this warning in that scenario.
So it now appears that your partition table(s) are wiped out? All of them, on all of the disks? That's just as mystifying to me as if entire disk PV's had been wiped out instead. But to be honest, a partition table wipe-out will probably be easier to deal with than and LVM wipeout. I'd say your chances of recovering a partition table are better.
Did you backup your partition table(s) before this disaster befell you? You might be able to use dd to extract your partition table to a simple file. It's only 16 bytes long (I think?) Very small, anyway. You could manually look analyze it byte-by-byte and see if it looks anywhere near sane (and repairable) or if it's total garbage. I used to know the exact makeup of a partition table off the top of my head, but I've long since forgotten the details. A Google search is your friend on this.
Distribution: debian on servers, ubuntu on desktops/laptops
Posts: 45
Original Poster
Rep:
Quote:
Originally Posted by haertig
So it now appears that your partition table(s) are wiped out?
Yep, that's what I was thinking...
Quote:
Originally Posted by akelder
RAID volumes are healthy according to /cat/proc/mdstat and mdadm, however fdisk claims there isn't a valid partition table on /dev/md1 or /dev/md0. Is it correct to say that might be the root
cause?
It's worth noting that the partition information is missing from /dev/md0 also, where my root fs lives (whichs is why I'm in no hurry to reboot) . The partition setup on /dev/md1 was simple; just one big partition which was used as a PV for LVM.
I usually save my partition and RAID setup info to help me out when disks fail, so I have to go dig that up.
haertig, I appreciate your time thinking about this problem. While stuff breaking is an inconvenience, this is when I find myself learning the most, so it's hard to complain..
Distribution: debian on servers, ubuntu on desktops/laptops
Posts: 45
Original Poster
Rep:
Quote:
Originally Posted by haertig
You might be able to use dd to extract your partition table to a simple file. It's only 16 bytes long (I think?)
Do you recall where on disk partition info is kept and where LVM keeps its stuff? What's a good utility to examine parts of disk that contain partition and LVM info?
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.