CentOS 6 libvirt KVM Live Snapshot is Slow
 

Hello Everyone,

I am currently evaluating KVM running on CentOS 6 (we are primarily a CentOS shop here so I figured I would keep the number of distributions we need to support down). I VT turned on in the BIOS (required for KVM) and installed CentOS with the Virtual Host option. I then imported the CentOS 6 Continuous Release repository to get the latest updates (snapshotting wasn't working at all before I did this, the error was "unable to execute QEMU command savevm").

I am using the virsh command line "virsh snapshot-create <domain>" to create the snapshot. The issue is that the snapshot is taking 10-20 minutes to create (the VM is suspended and unavailable for use during the time the snapshot is being taken), however the snapshot appears to be working. My question really is, is this normal behavior? Coming from a VMware background where snapshots are fast this seems very odd that it would take this long to generate a live snapshot. I just have a feeling I am missing something, but not sure what as I am using mostly defaults.

Here are my host specs:
* Dell PowerEdge R310
* 12 GB of RAM
* 250 GB hard drive RAID 1 (this is not yet designed for production)
* One 2.6 GHz Intel Xeon Quad Core Processor

Here are my guest specs (note there is only one guest running for now)
* Used Virtual Machine Manager GUI to create the VM
* CentOS 5.6 base install (from ISO)
* Disk Cache = none
* Hard Drive Size 20GB qcow2 preallocated format (using the qemu-img command, Virt-Manager appears to only create RAW images, not capable of snapshotting)
* 1024 MB of RAM
* Defined the VM as RHEL 5.4 or Greater

Here is the time it took for a snapshot today:

[root@server1 ~]# time virsh snapshot-create test-vm
Domain snapshot 1320175789 created


real 18m59.456s
user 0m0.004s
sys 0m0.010s

Here are my versions:
[root@server1 ~]# libvirtd --version
libvirtd (libvirt) 0.8.7

virsh # version
Compiled against library: libvir 0.8.7
Using library: libvir 0.8.7
Using API: QEMU 0.8.7
Running hypervisor: QEMU 0.12.1

[root@server1 ~]# uname -a
Linux server1 2.6.32-131.17.1.el6.x86_64 #1 SMP Thu Oct 6 19:24:09 BST 2011 x86_64 x86_64 x86_64 GNU/Linux

[root@server1 ~]# cat /etc/redhat-release
CentOS Linux release 6.0 (Final)

I will be happy to provide any other information you feel is needed and I am continuing my testing and research on this issue.

Thanks

first of all, have you tested how long it takes to snapshot a VM that isn't running? savevm does two things - suspend the VM (miggrate to file essentially) and then takes the snapshot, replaces the image used by the VM with the new snapshot, and resumes the VM.

If you try to make a snapshot when the VM is down, you eliminate the first part, and if that is still slow, you can see that the snapshot itself is slow. If it's fast, then you have something going on int he migration to file process, and not in the snapshot creation. Either way, it's a step in the right direction

The snapshot time while the VM is off is very fast, faster than I would expect the online snapshot to be.

[root@server1 ~]# time virsh snapshot-create test-vm
Domain snapshot 1320267158 created


real 0m1.162s
user 0m0.001s
sys 0m0.003s

The VM has 1024 MB of RAM I would assume since it is freezing the VM that it is dumping that memory to a file?

yes it is. is the VM busy when you run savevm?

No the system is idle, the guest is a test CentOS 5.6 i386 machine. It is just started up.

definitely something wrong with the migration... what is your /var mounted on?

/var is mounted root (I don't have a specific mount point for it), meaning it's on the RAID 1. I also have an R410 with 12 GB of RAM and 4 x 500 GB hard drives in a RAID 10 that is exhibiting the same problem with savevm.

I found this ticket online https://bugs.launchpad.net/qemu/+bug/741887 for QEMU, it appears to be my problem. However changing the cache to unsafe does not appear to have any effect (I changed it in the VM XML).

dyasny, are you using KVM with live snapshots working quickly using libvirt?

after you updated the xml, did you make sure it worked? restart vm, get it's cmdline (ps -ef|grep qemu) and see if the drive is started with cache=writeback

I do not use live snapshots, since it is not a supported feature yet. I do however understand that your issue is not really with the snapshot part, but rather with the VM suspend flow.

At this point, I'd probably start asking questions at the kvm or qemu IRC channels on freeNode, or email the mailing lists for help, since debugging the way kvm manipulates VM memory can be a tricky task