Debian Lenny - SAN - LVM fail
(This was crossposted from serverfault.com because no answer after several days)
I've got a Lenny server that has got a SAN connection configured as the only PV for a VG named 'datavg'.
Yesterday, I've updated the box with Debian patches and gave it a reboot.
After the reboot, it didn't boot up saying that it couldnt find /dev/mapper/datavg-datalv.
This is what I did:
- booted in rescue-mode and commented the mount in /etc/fstab
- rebooted into full-user mode. (mountpoint is /data, only postgresql could not start)
- did vgdisplay, lvdisplay, pvdisplay to find out what happened to the volume group. (datavg was missing entirely)
After that, I noticed that the LUN is visible from Linux and that the LVM partition is also visible:
- I ran pvck on the partition, but it did not find any label:
- I decided to write a new LVM label to the partition and restore the parameters from the backup file.
- After that, I appears when I issue a pvscan.
- With a vgchange -ay datavg, I activated the VG and the LV came available.
- When I tried to mount the LV, it did not find any filesystem. I tried recovery in several ways, but did not succeed.
- After making a DD of the affected LV, I've tried to recreate the superblocks with
- How can it be that the LVM label disappears after patches and a reboot?
- Why is the filesystem not there after salvaging the PV? (Did the pvcreate command trash the data?)
- Is the ext3 filesystem in the LV still salvageable?
- Is there anything I could have done to prevent this issue?
Thanks in advance,
Just re-read your question, so I'm changing my post. I'm wondering if the problem is that you tried to restore the label before running vgchange -ay.
vgchange should find the physical volumes, volume groups, etc, and add them to the local system, but I don't believe it changes them at all. The label didn't disappear, your system just didn't have it yet. I'd guess it was the attempts to restore/recreate that blew away some of the data, but again, it's a guess.
I've mounted an iSCSI volume with LVM on it from multiple boxes, re-installs, etc, and vgchange -ay was all I needed to get the volume groups on that system.
Your SAN volume, and the LVM on it shouldn't be affected at all by an OS update, so you're not crazy for thinking this is weird.
Note, my san skills are still building, I'm very far from expert! I'm in no way positive how to recover the data in your situation, but I'm hoping you post it (if you're able to), in case I ever run into the same.
I had to manually insert the label in the PV to get it recognized. (see the pvcreate command)
|All times are GMT -5. The time now is 10:21 PM.|