Libvirt Virtual Machine not being LIVE Migrated by pcs using ocf:heartbeat:VirtualDomain

bulletproof.rs · 06-25-2019, 11:21 AM

Hello everyone,

We have just built a KVM cluster on some ProLiant DL560's on top ov CentOS 7.
We have created GFS2 cluster using pacemaker/corosync and installed libvirt (4.5.0) on them.
Beside GFS2 cluster, servers are connected to NetApp storage and we map NetApp LUN's to all hypervisors. Those LUN's are to be used on Virtual Machines created in libvirt/qemu.

To clusterize everything and minimize all potential downtime, we wanted pacemaker/corosync to manage virtual machines.
Now here comes the fun (annoying) part.

- I create a VPS Server and when defining disk, i will point to mapped LUN (/dev/mapper/proba). Disk is define without cache (cache='none')
- I shut it down, edit it's xml file in /etc/libvirt/qemu so that i remove "CPU" directive.
- I Copy that xml to shared GFS2 directory /shared/xml
- I undefine VPS (let's say proba) using "virsh undefine proba"
- Now i create a pcs resource using following command

Code:

resource create proba_vps VirtualDomain hypervisor="qemu:///system" config="/shared/xml/proba.xml" migration_transport=ssh op start timeout="300s" op stop timeout="300s" op monitor timeout="30" interval="10"  meta allow-migrate="true" priority="100" target-role="Started" is-managed="true" op migrate_from interval="0" timeout="120s" op migrate_to interval="0" timeout="120" --group AQUILA

It will successfully define it and automatically start it. But the problem lies when i try to move that to another hypervisor using the command

Code:

pcs resource move proba kvm_aquila-02

It will move it, but it will shut it down first and then move it, but instead id should perform LIVE seamless migration.
If i create the same VPS but one using QCOW image instead of multipath LUN, it will perform live migration.

I don't know if the problem lies in pacemaker/corosync or libvirt itself, so i would like to hear some thoughts to get this resolved.

If i define VPS in plain libvirt, without pacemaker/corosync and try to migrate it using

Code:

virsh migrate proba qemu+ssh://kvm_aquila-02/system --live --persistent --undefinesource

It will not allow me to do so, but will error out with

Code:

error: Unsafe migration: Migration without shared storage is unsafe

If i add "--unsafe" flag it will pass fine, without errors, issues etc...

Maybe pacemaker/corosync is doing the same, trying live migration without --unsafe. Maybe problem lies somewhere else. Don't know.

We have plenty of hypervisors on top of CentOS 6 and libvirt 0.10.2 and migration with virsh migrate with same parameters posted above works without issues and errors.

What to do?

Thanks in advance,
Marko Todoric

bulletproof.rs · 06-26-2019, 03:46 AM

So i found out why is this happening. Namely I've tried updating libvirt and all it's components to version 4.10.0, but still the same issue happened. I've reviewed xml files between CentOS 6 hypervisors and new one on CentOS 7. And basically this is the difference when i comes to disk setup:

CentOS 6 (Libvirt 0.10.2)

Code:

    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/mapper/tables1'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>

CentOS 7 (Libvirt 4.10.0)

Code:

    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/dev/mapper/proba'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>

So basically, even if i put disk as a path to block device, libvirt will recognize it as a "file" instead of "block" device. When i changed the xml file and tried to migrate with options mentioned in previous post - all worked well!

Now, could this be a bug in libvirt causing it not to recognize block device or am i doing something wrong? We have been creating VM's in CentOS 7 the same way we did in CentOS 6 hypervisors, and this is the way:

virt-install --connect=qemu+ssh://kvm_aquila-01/system \
--name=proba \
--hvm \
--graphics vnc \
--disk path=/dev/mapper/proba,bus=virtio,cache=none \
--vcpus 2 \
--ram=2048 \
--location=http://192.168.54.182/mirror \
--extra-args "ksdevice=eth0 inst.ks=http://192.168.53.99/ks/proba.cfg" \
--network bridge=br7,model=virtio \
--boot hd,cdrom,menu=on \
--os-type=linux \
--os-variant=rhel7.0 \
--metadata description="LinuxTim - Proba" \
--noreboot

How do we make libvirt automatically default to block device instead of file when it comes to disk type?
Is this even possible ?