here at work we have a few virtual servers, running on Proxmox. There are nightly backups scheduled for each of them, through Proxmox itself.
One day, backups on one physical node started failing with the following error:
command 'lvcreate --size 1024M --snapshot --name 'vzsnap-ve2-0' '/dev/ve2ds1-sync/vm-201-disk-1'' failed: exit code 5
Note: /dev/ve2ds1-sync is an LVM VG that is replicated on another machine through drbd for redundancy. vm-201-disk-1 is the LV that is used as virtual disk for one of the virtual machines, but the same happens for all VGs.
Some more info is available in the logs:
Mar 11 01:00:03 INFO: device-mapper: create ioctl on ve2ds1--sync-vzsnap--ve2--0 failed: Device or resource busy
Mar 11 01:00:03 INFO: Aborting. Failed to activate snapshot exception store.
So it seems LVM cannot create LVs anymore. dmesg reports a few errors:
device-mapper: ioctl: device doesn't appear to be in the dev hash table.
device-mapper: table: 253:9: linear: dm-linear: Device lookup failed
device-mapper: ioctl: error adding target to table
The LV shows up in dmsetup ls:
root@ve2:~# dmsetup ls
I can remove it with dmsetup remove but this doesn't solve anything. What can I do? I might try rebooting the server, but that is really the last resort, since these servers are production ones.
Any suggestions? Thanks in advance!