dm-linear: Device lookup failed

SukkoPera · 04-11-2013, 02:16 AM

Hi everybody,

here at work we have a few virtual servers, running on Proxmox. There are nightly backups scheduled for each of them, through Proxmox itself.

One day, backups on one physical node started failing with the following error:

Code:

command 'lvcreate --size 1024M --snapshot --name 'vzsnap-ve2-0' '/dev/ve2ds1-sync/vm-201-disk-1'' failed: exit code 5

Note: /dev/ve2ds1-sync is an LVM VG that is replicated on another machine through drbd for redundancy. vm-201-disk-1 is the LV that is used as virtual disk for one of the virtual machines, but the same happens for all VGs.

Some more info is available in the logs:

Code:

Mar 11 01:00:03 INFO:   device-mapper: create ioctl on ve2ds1--sync-vzsnap--ve2--0 failed: Device or resource busy
Mar 11 01:00:03 INFO:   Aborting. Failed to activate snapshot exception store.

So it seems LVM cannot create LVs anymore. dmesg reports a few errors:

Code:

device-mapper: ioctl: device doesn't appear to be in the dev hash table.
device-mapper: table: 253:9: linear: dm-linear: Device lookup failed
device-mapper: ioctl: error adding target to table

The LV shows up in dmsetup ls:

Code:

root@ve2:~# dmsetup ls
ve2ds1--sync-vzsnap--ve2--0     (253:9)

I can remove it with dmsetup remove but this doesn't solve anything. What can I do? I might try rebooting the server, but that is really the last resort, since these servers are production ones.

Any suggestions? Thanks in advance!

smallpond · 04-12-2013, 12:56 PM

Exit code 5 probably means EIO - an I/O error occurred.
Check the status of your LVM with

Code:

lvdisplay /dev/ve2ds1-sync/vm-201-disk-1

Also check your log for disk errors.

SukkoPera · 04-15-2013, 02:41 AM

Thanks for the help, smallpond. The LVM is ok, AFAIK:

Code:

root@ve2:~# lvdisplay /dev/ve2ds1-sync/vm-201-disk-1
  --- Logical volume ---
  LV Path                /dev/ve2ds1-sync/vm-201-disk-1
  LV Name                vm-201-disk-1
  VG Name                ve2ds1-sync
  LV UUID                v4HJgO-ce1E-uEtm-OlKH-149O-PsGK-TRnnTq
  LV Write Access        read/write
  LV Creation host, time ve2, 2012-12-02 17:28:10 +0100
  LV Status              available
  # open                 1
  LV Size                20.00 GiB
  Current LE             5120
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:5

Also, the disks are OK, as they are hardware RAID1 (MegaRAID) and we have no failures ATM.

whizje · 04-15-2013, 03:06 AM

test the disks with smartctl -t long /dev/sda

SukkoPera · 04-15-2013, 06:58 AM

I can't use smartctl, as my disks are logical disks and the physical disks are hidden by the MegaRAID controller. Anyway I have started a RAID consistency check with MegaRAID's utility, and all my disks are OK.

SukkoPera · 05-08-2013, 07:20 AM

Just for reference: we finally solved this rebooting everything.