[SOLVED] CURRENT: LVM/CRYPTSETUP duplicate devices in /dev/mapper again.

GazL · 01-23-2011, 10:55 AM

We had this problem a while back but it went away with whatever updates lvm/cryptsetup have been through, but it's back again as of the latest update.

Instead of symlinks to the /dev/dm-* devices, lvm and cryptsetup are both creating duplicate device nodes under /dev/mapper for already existing devices.

e.g.

Code:

gazl@slack:~$ ls -l /dev/mapper
total 0
crw------- 1 root root  10, 61 Jan 23 16:48 control
brw-rw-r-- 1 root disk 253,  0 Jan 23 16:48 lukssda5
brw-rw-r-- 1 root disk 253,  9 Jan 23 16:48 rootvg-lvhome
brw-rw-r-- 1 root disk 253, 10 Jan 23 16:48 rootvg-lvlocal
brw-rw-r-- 1 root disk 253,  6 Jan 23 16:48 rootvg-lvroot
brw-rw-r-- 1 root disk 253,  8 Jan 23 16:48 rootvg-lvtmp
brw-rw-r-- 1 root disk 253,  7 Jan 23 16:48 rootvg-lvvar
brw-rw-r-- 1 root disk 253,  4 Jan 23 16:48 vgsys14-lvhome
brw-rw-r-- 1 root disk 253,  1 Jan 23 16:48 vgsys14-lvroot
brw-rw-r-- 1 root disk 253,  2 Jan 23 16:48 vgsys14-lvswap01
brw-rw-r-- 1 root disk 253,  5 Jan 23 16:48 vgsys14-lvtmp
brw-rw-r-- 1 root disk 253,  3 Jan 23 16:48 vgsys14-lvvar
gazl@slack:~$ ls -l /dev/dm-*
brw-rw---- 1 root disk 253,  0 Jan 23 16:48 /dev/dm-0
brw-rw---- 1 root disk 253,  1 Jan 23 16:48 /dev/dm-1
brw-rw---- 1 root disk 253, 10 Jan 23 16:48 /dev/dm-10
brw-rw---- 1 root disk 253,  2 Jan 23 16:48 /dev/dm-2
brw-rw---- 1 root disk 253,  3 Jan 23 16:48 /dev/dm-3
brw-rw---- 1 root disk 253,  4 Jan 23 16:48 /dev/dm-4
brw-rw---- 1 root disk 253,  5 Jan 23 16:48 /dev/dm-5
brw-rw---- 1 root disk 253,  6 Jan 23 16:48 /dev/dm-6
brw-rw---- 1 root disk 253,  7 Jan 23 16:48 /dev/dm-7
brw-rw---- 1 root disk 253,  8 Jan 23 16:48 /dev/dm-8
brw-rw---- 1 root disk 253,  9 Jan 23 16:48 /dev/dm-9

P.S. Also, note the 'other' permissions!!!

GazL · 01-23-2011, 11:11 AM

Tried reverting latest udev, doesn't seem to be the issue.
Tried reverting lvm to 79-1 but it's still the same.

It looks like this might date back a little further than the latest set of updates and I just hadn't noticed.
I don't have any packages going any further back that this so I can't test any older ones.

Darth Vader · 01-23-2011, 01:30 PM

It's a big WTF moment, right? It happened to me, too ...

So what's the problem? In fact, /dev filesystem is started early into initrd and moved to real /dev. If the initrd utilities are not the same as into system, is going right into these duplicate devices, some times.

My advice is to use udev option in the initrd and regenerate it, if you upgrade your sensitive packages, into the system (LVM, cryptsetup, mdadm).

GazL · 01-23-2011, 02:16 PM

Thanks for the reply Darth.

I'd already rebuilt my initrd, so the versions should all have been in sync, but on your suggestion I've added the -u option to the mkinitrd command (although I was under the impression that the initrd already used udev/devtmpfs by defaut?) and the luks entry under /dev/mapper is now back to a symlink but it didn't change the devices for the lvm logical volumes. Also, I now get a bunch of "semop failed for cookie" messages during the cryptsetup unlock which is a little worrying.

Darth Vader · 01-23-2011, 07:23 PM

Quote:

Originally Posted by GazL

Thanks for the reply Darth.

I'd already rebuilt my initrd, so the versions should all have been in sync, but on your suggestion I've added the -u option to the mkinitrd command (although I was under the impression that the initrd already used udev/devtmpfs by defaut?) and the luks entry under /dev/mapper is now back to a symlink but it didn't change the devices for the lvm logical volumes. Also, I now get a bunch of "semop failed for cookie" messages during the cryptsetup unlock which is a little worrying.

How about a little cryptsetup downgrade?

GazL · 01-24-2011, 04:32 AM

Quote:

Originally Posted by Darth Vader

How about a little cryptsetup downgrade?

Won't help (lvm is also affected so cryptsetup is just a victim), This looks to be a device-mapper/udev interaction type scenario to me. Perhaps going back to udev-164 might sort it. I'll investigate a little more when I have the time.

conraid · 01-24-2011, 05:52 AM

Code:

$ ls -l
totale 0
brw-rw-r-- 1 root disk 254,   1 gen 24 10:43 blanklvm-home
brw-rw-r-- 1 root disk 254,   3 gen 24 10:43 blanklvm-local
brw-rw-r-- 1 root disk 254,   2 gen 24 10:43 blanklvm-opt
brw-rw-r-- 1 root disk 254,   0 gen 24 11:43 blanklvm-root
crw------- 1 root root  10, 236 gen 24 11:43 control
lrwxrwxrwx 1 root root        7 gen 24 11:43 cryptlvm-dati -> ../dm-5
lrwxrwxrwx 1 root root        7 gen 24 11:43 cryptlvm-swap -> ../dm-6
lrwxrwxrwx 1 root root        7 gen 24 11:43 cryptsda2 -> ../dm-4

$ ls -l dm-*
brw-rw---- 1 root disk 254, 0 gen 24 11:43 dm-0
brw-rw---- 1 root disk 254, 1 gen 24 11:43 dm-1
brw-rw---- 1 root disk 254, 2 gen 24 11:43 dm-2
brw-rw---- 1 root disk 254, 3 gen 24 11:43 dm-3
brw-rw---- 1 root disk 254, 4 gen 24 11:43 dm-4
brw-rw---- 1 root disk 254, 5 gen 24 10:43 dm-5
brw-rw---- 1 root disk 254, 6 gen 24 11:43 dm-6

cryptsetup not duplicate device

GazL · 01-24-2011, 08:03 AM

Quote:

Originally Posted by conraid

Code:

$ ls -l
totale 0
brw-rw-r-- 1 root disk 254,   1 gen 24 10:43 blanklvm-home
brw-rw-r-- 1 root disk 254,   3 gen 24 10:43 blanklvm-local
brw-rw-r-- 1 root disk 254,   2 gen 24 10:43 blanklvm-opt
brw-rw-r-- 1 root disk 254,   0 gen 24 11:43 blanklvm-root
crw------- 1 root root  10, 236 gen 24 11:43 control
lrwxrwxrwx 1 root root        7 gen 24 11:43 cryptlvm-dati -> ../dm-5
lrwxrwxrwx 1 root root        7 gen 24 11:43 cryptlvm-swap -> ../dm-6
lrwxrwxrwx 1 root root        7 gen 24 11:43 cryptsda2 -> ../dm-4

$ ls -l dm-*
brw-rw---- 1 root disk 254, 0 gen 24 11:43 dm-0
brw-rw---- 1 root disk 254, 1 gen 24 11:43 dm-1
brw-rw---- 1 root disk 254, 2 gen 24 11:43 dm-2
brw-rw---- 1 root disk 254, 3 gen 24 11:43 dm-3
brw-rw---- 1 root disk 254, 4 gen 24 11:43 dm-4
brw-rw---- 1 root disk 254, 5 gen 24 10:43 dm-5
brw-rw---- 1 root disk 254, 6 gen 24 11:43 dm-6

cryptsetup not duplicate device

No, but lvm is. Those should also be symlinks to /dev/dm-*

conraid · 01-24-2011, 08:23 AM

Quote:

Originally Posted by GazL

No, but lvm is. Those should also be symlinks to /dev/dm-*

yes, I just wrote to confirm your "lvm is also affected so cryptsetup is just a victim"

GazL · 01-24-2011, 11:35 AM

Quote:

Originally Posted by conraid

yes, I just wrote to confirm your "lvm is also affected so cryptsetup is just a victim"

Ahh, sorry I misunderstood then. Thankyou for clarifying.

conraid · 01-25-2011, 03:21 AM

Quote:

Originally Posted by GazL

Ahh, sorry I misunderstood then. Thankyou for clarifying.

my English is bad, and sometimes I avoid writing :-)

GazL · 01-31-2011, 06:36 PM

Just got to the bottom of this. Turned out to be a sneaky one...
udevd is being killed by the initrd's init script before it's finished processing all the lvm events.

Potential fix mailed to Pat.

Still getting the semop failed message from cryptsetup, but it doesn't appear to be fatal and I think it's unrelated.

I'll mark this as 'solved'