[SOLVED] Logical volume not available

citocran · 08-13-2011, 03:46 PM

Hi there,

after hours surfing through forums to find the answer to my question I'm desperate now.

I've an Kubuntu 11.04 on x64. My hard drive is encrypted (swap/home/root).

The boot process is normal and I enter my HD-password. Than Kubuntu won't load. I get error messages like:

Quote:

udevd[65]: worker [278] unexpectedly returned with status 0x0100
udevd[65]: worker [278] failed while handling ' /devices/virtual/block/dm-0'
udevd[65]: worker [157] failed while handling ' /devices/virtual/block/dm-1'
/dev/dm-1/: read failed after 0 of 4096 at 0: Input/Output error
...

So, there seems to be a problem with the encrypted HD. With an Ubuntu running on an USB-stick I tried to get closer to this problem.
In a terminal I can open the volume group containing /swap /home and /root. Even after vgchange -ay and lvchange -ay only the /swap is available.
Output of lvdisplay is:

Quote:

/dev/mapper/lvm-root: open failed: No such file or directory
--- Logical volume ---
LV Name /dev/lvm/root
VG Name lvm
LV UUID KoHUPE-i8T3-oAvw-dmIx-fEvy-gSpH-JUtYXe
LV Write Access read/write
LV Status NOT available
LV Size 27,94 GiB
Current LE 7152
Segments 1
Allocation inherit
Read ahead sectors auto

--- Logical volume ---
LV Name /dev/lvm/swap
VG Name lvm
LV UUID laiW4r-btGu-3qN7-CeYU-i9Kr-lrZi-uXpuTO
LV Write Access read/write
LV Status available
# open 0
LV Size 1,86 GiB
Current LE 476
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 252:2

/dev/mapper/lvm-home: open failed: No such file or directory
--- Logical volume ---
LV Name /dev/lvm/home
VG Name lvm
LV UUID fsu97O-euX0-2ZDf-lqET-66u8-edqX-01Dhut
LV Write Access read/write
LV Status NOT available
LV Size 202,85 GiB
Current LE 51929
Segments 1
Allocation inherit
Read ahead sectors auto

Any ideas how to make the LV's available?
I think the easiest solution would be a new installation but I've some data and settings I don't want to lose.

I'd appreciate any new approach!

crts · 08-14-2011, 10:45 AM

Hi,

so do I understand you correctly that your OS on the USB stick functions as rescue OS?
When you tried to mount them manually did you succeed to decrypt them properly first?
It is strange that it would only recognize the swap partition. What does your exact setup look like?
Is it one encrypted, physical hdd with an lvm overlay or is it an lvm where the each logical volume is encrypted separately?
If we do not know how your system is setup then we cannot suggest appropriate measures for recovery.
So please answer above questions first.

onebuck · 08-14-2011, 11:00 AM

Moved: This thread is more suitable in <Ubuntu> and has been moved accordingly to help your thread/question get the exposure it deserves.

citocran · 08-14-2011, 11:41 AM

Quote:

so do I understand you correctly that your OS on the USB stick functions as rescue OS?

It's a live version of Ubuntu. Not really a rescue OS.

Quote:

When you tried to mount them manually did you succeed to decrypt them properly first?

What I tried with the help if the USB OS is:

Code:

mkdir /media/test
modprobe dm-crypt
cryptsetup luksOpen /dev/sda5 test
vgchange -ay
mount /dev/lvm/home /media/test

The decryption works. The vgchange takes very long or never finishes at all. But when I do it twice I get the output "3 logical volume(s) in volume group "lvm" now active". When I try to mount /home or /root I get the error "mount: special device /dev/lvm/home does not exist". At this point I did lvdisplay (see above).

Quote:

It is strange that it would only recognize the swap partition. What does your exact setup look like?
Is it one encrypted, physical hdd with an lvm overlay or is it an lvm where the each logical volume is encrypted separately?

Is it one encrypted, physical hdd with an lvm overlay.

GazL · 08-14-2011, 01:27 PM

You might want to try adding a "vgscan --mknodes" after the cryptsetup and before the "vgchange -ay" and see if that helps. Normally it shouldn't be necessary, but I've had rare occasions when they weren't created for some reason.

crts · 08-14-2011, 01:45 PM

Quote:

Originally Posted by citocran

The vgchange takes very long or never finishes at all. But when I do it twice I get the output "3 logical volume(s) in volume group "lvm" now active".

Normally it is just a matter of seconds until it finds the volume groups. Did you do anything with the lvm before this behavior started? Like trying to expand/modify it in any way?

Can you post the output of:

Code:

ls -l /dev/mapper/*
ls -l /dev/dm-*

Issue above commands after you applied
vgchange -a y

citocran · 08-14-2011, 04:56 PM

Code:

vgscan --mknodes

Worked so far. All three volumes are available. Now I'm waiting since 15 minutes that they're getting mounted. Don't know why some commands take such a long time.
So, what does that mean? Can I fix this somehow?

output of

Code:

ls -l /dev/mapper/*
ls -l /dev/dm-*

is

Quote:

ubuntu@ubuntu:/dev/lvm$ ls -l /dev/mapper/*
crw------- 1 root root 10, 236 2011-08-14 15:56 /dev/mapper/control
brw-rw---- 1 root disk 252, 3 2011-08-14 21:22 /dev/mapper/lvm-home
brw-rw---- 1 root disk 252, 1 2011-08-14 21:22 /dev/mapper/lvm-root
lrwxrwxrwx 1 root root 7 2011-08-14 16:39 /dev/mapper/lvm-swap -> ../dm-2
lrwxrwxrwx 1 root root 7 2011-08-14 16:14 /dev/mapper/test -> ../dm-0
lrwxrwxrwx 1 root root 7 2011-08-14 21:28 /dev/mapper/udisks-luks-uuid-6c19ae0f-f2fa-438d-85bf-7954672113dc-uid999 -> ../dm-4
ubuntu@ubuntu:/dev/lvm$ ls -l /dev/dm-*
brw-rw---- 1 root disk 252, 0 2011-08-14 16:14 /dev/dm-0
brw------- 1 root root 252, 1 2011-08-14 16:14 /dev/dm-1
brw-rw---- 1 root disk 252, 2 2011-08-14 16:39 /dev/dm-2
brw------- 1 root root 252, 3 2011-08-14 16:36 /dev/dm-3
brw-rw---- 1 root disk 252, 4 2011-08-14 21:28 /dev/dm-4

I've nothing done before concerning the volume groups. Everything was the same for over four months.

crts · 08-14-2011, 05:46 PM

How exactly are you mounting them now?

Code:

ubuntu@ubuntu:/dev/lvm$ ls -l /dev/mapper/*
crw------- 1 root root 10, 236 2011-08-14 15:56 /dev/mapper/control
brw-rw---- 1 root disk 252, 3 2011-08-14 21:22 /dev/mapper/lvm-home
brw-rw---- 1 root disk 252, 1 2011-08-14 21:22 /dev/mapper/lvm-root
lrwxrwxrwx 1 root root 7 2011-08-14 16:39 /dev/mapper/lvm-swap -> ../dm-2
lrwxrwxrwx 1 root root 7 2011-08-14 16:14 /dev/mapper/test -> ../dm-0
lrwxrwxrwx 1 root root 7 2011-08-14 21:28 /dev/mapper/udisks-luks-uuid-6c19ae0f-f2fa-438d-85bf-7954672113dc-uid999 -> ../dm-4
ubuntu@ubuntu:/dev/lvm$ ls -l /dev/dm-*
brw-rw---- 1 root disk 252, 0 2011-08-14 16:14 /dev/dm-0
brw------- 1 root root 252, 1 2011-08-14 16:14 /dev/dm-1
brw-rw---- 1 root disk 252, 2 2011-08-14 16:39 /dev/dm-2
brw------- 1 root root 252, 3 2011-08-14 16:36 /dev/dm-3
brw-rw---- 1 root disk 252, 4 2011-08-14 21:28 /dev/dm-4

Normally /dev/mapper/lvm-home should be a link to /dev/dm-3 or so. I suspect that issuing
vgscan --mknodes

resulted in /dev/mapper/lvm-home being the block device instead of just a link. Ubuntu probably relies on it being a link which might be causing the trouble with mount. Try mounting it in different ways:

Code:

mount /dev/mapper/lvm-home /mnt
# or
mount /dev/dm-3 /mnt

I also see that /dev/mapper/test is mapped to /dev/dm-0. Never saw that one before.
Also, /dev/mapper/udisks-luks-uuid-61... seems to be your encrypted partition. Since it has to be decrypted before lvm gets active I would assume that it should be mapped to /dev/dm-0 or /dev/dm-1. Yet it is being mapped to /dev/dm-4. This also looks strange.
Reboot into your rescue OS and post the output of

Code:

fdisk -l

Decrypt your partition, then issue

Code:

vgchange -a y

but this time without executing 'vgscan --mknodes'. I'd like to see how it is handled the default way.

Quote:

Now I'm waiting since 15 minutes that they're getting mounted.

As I said this might be related to 'vgscan --mknodes', not sure. Worst case scenario is that there might be a hardware failure involved. This would minimize the chances of a successful recovery.
Do you have a spare disk to make an image of the partition? I was using ddrescue of gddrescue in case of faulty hardware.

GazL · 08-14-2011, 05:59 PM

It really shouldn't be taking that long to mount the filesystems. Are you seeing any errors/timeouts listed in the output of 'dmesg' that might indicate hardware issues?

Next step might be trying to mount them as read-only, and if you can, copy your data to another disk for safe keeping

After that you might want to try a fsck (e.g. fsck /dev/lvm/root) while the filesystem is unmounted to see if you can repair things and get things back to normal.

GazL · 08-14-2011, 06:12 PM

Quote:

Originally Posted by crts

As I said this might be related to 'vgscan --mknodes', not sure.

The creation of the duplicate device nodes in /dev/mapper rather than creation of symlinks to the /dev/dm-* device nodes is something I've seen before in both Slackware and Fedora with certain combinations of udev, dev-mapper and cryptsetup versions. Ideally you'd see the symlinks, but the duplicate devices will have the correct device major and minor numbers so it won't hurt. It may just be that the versions of those components included in the Ubuntu version he is using are affected by that same issue.

citocran · 08-14-2011, 06:51 PM

I tried different versions of mounting. In general I get the message that I have to specify the filesystem type. If I do so I get

Quote:

mount: wrong fs type, bad option, bad superblock on /dev/mapper/lvm-root,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so

dmesg | tail gives me:

Quote:

ubuntu@ubuntu:/dev/lvm$ dmesg|tail
[24822.084789] sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[24822.084798] sd 0:0:0:0: [sda] Sense Key : Medium Error [current] [descriptor]
[24822.084808] Descriptor sense data with sense descriptors (in hex):
[24822.084813] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[24822.084834] 00 07 b1 8a
[24822.084844] sd 0:0:0:0: [sda] Add. Sense: Unrecovered read error - auto reallocate failed
[24822.084854] sd 0:0:0:0: [sda] CDB: Read(10): 28 00 00 07 b1 8a 00 00 02 00
[24822.084874] end_request: I/O error, dev sda, sector 504202
[24822.084939] ata1: EH complete
[24822.084945] EXT4-fs (dm-1): unable to read s

I'm traveling right now and of course I have no additional HD with that would be big enough to fit all the data on. First thing I'll do tomorrow morning is to buy one.

Now I'll reboot and do what crts had asked for. Sorry it just takes so much time to wait for the commands to be executed. There's definitely something wrong

citocran · 08-14-2011, 07:00 PM

output of fdsik -l after reboot:

Quote:

Disk /dev/sda: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0004dc19

Device Boot Start End Blocks Id System
/dev/sda1 * 1 32 248832 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 32 30402 243947521 5 Extended
/dev/sda5 32 30402 243947520 83 Linux

Disk /dev/sdb: 8189 MB, 8189378560 bytes
252 heads, 62 sectors/track, 1023 cylinders
Units = cylinders of 15624 * 512 = 7999488 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000217fc

Device Boot Start End Blocks Id System
/dev/sdb1 * 1 1023 7991645 c W95 FAT32 (LBA)
Partition 1 has different physical/logical endings:
phys=(1023, 251, 62) logical=(1022, 251, 62)

crts · 08-14-2011, 07:01 PM

Quote:

Originally Posted by citocran

First thing I'll do tomorrow morning is to buy one.

You will only need a new one IF there is a hardware failure. Before you do that you could run some tests. My BIOS, e.g. can run some tests on the installed hardware. Maybe you should check this possibility before spending money on a new harddisk. On the other hand, if it is faulty then using it, even when just running some tests, might damage it further. So use your own best judgment since we cannot assist you with this decision.

citocran · 08-14-2011, 07:36 PM

Even the decryption takes now a long long time. I'm waiting for about 30 mins and not finished. Think I'll stop for today and try to rescue the data first before trying anything else.
Thanks so far!

jschiwal · 08-14-2011, 07:56 PM

You can use 'file -s <devnode>' to determine if a file system is present and help determine what mount options to use. I agree that you may be looking at a problem with the drive, cable or controller. This probably caused time outs unlocking a partition, or mapping a virtual partition. If you can get /home mounted, you may want to back up your data before reformatting or replacing the drive. Also check the kernel messages. I/O errors would indicate a drive may be going bad.