[SOLVED] lousy qemu-img performance converting to qcow2 ... if you use a hard drive

Skaperen · 05-25-2011, 12:27 PM

When using qemu-img to convert a raw disk image (file) into the qcow2 format, if the target output device (where the qcow2 file will be stored), the performance comes up lousy. This can be worked around using a tmpfs filesystem to cache the output file (then copy it to permanent disk storage), if you have enough RAM to hold the file in tmpfs.

Using strace, I see that qemu-img is doing way too many calls to fdatasync(). It seems there is one for each write() call. I'm assuming that qemu and qemu-img are using the same (library) code to implement qcow2, and that this code is the culprit in doing all the fdatasync() calls. I suspect the problems are the result of both the fdatasync() calls and the fact that many blocks are rewritten many times for the qcow2 format.

I did some benchmarks with an 8 GB image file I used to do a test install of Fedora-15 (x86_64) that just came out. I ran the install and a couple reboots of installed system under qemu, but using the raw image format in tmpfs. I then saved this image and proceeded with some benchmark tests.

1. Starting with an empty freshly mounted tmpfs, and the raw image file on disk, I ran qemu-img to convert it to qcow2 format in tmpfs, and copied the qcow2 file from tmpfs to disk, and did a sync command. I timed everything together, from the mounting of tmpfs, to copying the result, and the sync command. I did this 4 times in sequence. The 1st took 61 seconds wallclock time. Then 2nd, 3rd, and 4th took 60 seconds.

2. Starting with an empty freshly mounted tmpfs, and the raw image file on disk, I copied the raw image to tmpfs (took 29 seconds), and converted it to qcow2 format naming a new disk file as the output. The conversion alone took 25m41s, then 25m45s, then 25m46s, then 26m1s.

In summary, if you need to convert to qcow2 format, and have enough RAM to do this, you should stage the file in tmpfs, first, then copy it to disk.

Test 1 results ...

Code:

lorentz/phil /home/phil 236> ( for n in 1 2 3 4 ; do echo ; echo ; time ( rm -fv test1.qcow2 && umount /mnt/vm && mount /mnt/vm && time qemu-img convert -f raw -O qcow2 fedora-15-amd64.img /mnt/vm/fedora-15-amd64.qcow2 && time cp -pv /mnt/vm/fedora-15-amd64.qcow2 test1.qcow2 && time sync && ls -dl test1.qcow2 ) ; done )


removed `test1.qcow2'
[[ 0m31s real 31.044 - user 1.800 - sys 7.800 - 30.92% ]]
`/mnt/vm/fedora-15-amd64.qcow2' -> `test1.qcow2'
[[ 0m20s real 20.148 - user 0.020 - sys 3.390 - 16.92% ]]
[[ 0m9s real 9.332 - user 0.000 - sys 0.000 - 0.00% ]]
-rw-r--r-- 1 phil phil 3352821760 2011-05-25 11:28 test1.qcow2
[[ 1m1s real 61.588 - user 1.820 - sys 12.050 - 22.52% ]]


removed `test1.qcow2'
[[ 0m30s real 30.502 - user 1.840 - sys 7.990 - 32.22% ]]
`/mnt/vm/fedora-15-amd64.qcow2' -> `test1.qcow2'
[[ 0m19s real 19.337 - user 0.000 - sys 3.570 - 18.46% ]]
[[ 0m9s real 9.613 - user 0.000 - sys 0.000 - 0.00% ]]
-rw-r--r-- 1 phil phil 3352821760 2011-05-25 11:29 test1.qcow2
[[ 1m0s real 60.517 - user 1.840 - sys 12.410 - 23.54% ]]


removed `test1.qcow2'
[[ 0m30s real 30.608 - user 1.870 - sys 7.760 - 31.46% ]]
`/mnt/vm/fedora-15-amd64.qcow2' -> `test1.qcow2'
[[ 0m19s real 19.536 - user 0.020 - sys 3.450 - 17.76% ]]
[[ 0m9s real 9.094 - user 0.000 - sys 0.000 - 0.00% ]]
-rw-r--r-- 1 phil phil 3352821760 2011-05-25 11:30 test1.qcow2
[[ 1m0s real 60.332 - user 1.890 - sys 12.050 - 23.10% ]]


removed `test1.qcow2'
[[ 0m30s real 30.572 - user 1.760 - sys 7.810 - 31.30% ]]
`/mnt/vm/fedora-15-amd64.qcow2' -> `test1.qcow2'
[[ 0m19s real 19.439 - user 0.010 - sys 3.390 - 17.49% ]]
[[ 0m9s real 9.607 - user 0.000 - sys 0.000 - 0.00% ]]
-rw-r--r-- 1 phil phil 3352821760 2011-05-25 11:31 test1.qcow2
[[ 1m0s real 60.695 - user 1.770 - sys 12.060 - 22.78% ]]
lorentz/phil /home/phil 237>

Test 2 results ...

Code:

lorentz/phil /home/phil 248> ( for n in 1 2 3 4 ; do echo ; echo ; time ( rm -fv test2.qcow2 && umount /mnt/vm && mount /mnt/vm && time cp -pv fedora-15-amd64.img /mnt/vm/fedora-15-amd64.img && time qemu-img convert -f raw -O qcow2 /mnt/vm/fedora-15-amd64.img test2.qcow2 && time sync && ls -dl test2.qcow2 ) ; done )


removed `test2.qcow2'
`fedora-15-amd64.img' -> `/mnt/vm/fedora-15-amd64.img'
[[ 0m29s real 29.755 - user 0.370 - sys 5.830 - 20.83% ]]
[[ 25m41s real 1541.455 - user 2.550 - sys 18.010 - 1.33% ]]
[[ 0m0s real 0.391 - user 0.000 - sys 0.000 - 0.00% ]]
-rw-r--r-- 1 phil phil 3352821760 2011-05-25 12:03 test2.qcow2
[[ 26m12s real 1572.516 - user 2.920 - sys 24.690 - 1.75% ]]


removed `test2.qcow2'
`fedora-15-amd64.img' -> `/mnt/vm/fedora-15-amd64.img'
[[ 0m29s real 29.630 - user 0.470 - sys 5.600 - 20.48% ]]
[[ 25m45s real 1545.210 - user 3.000 - sys 17.830 - 1.34% ]]
[[ 0m0s real 0.533 - user 0.000 - sys 0.000 - 0.00% ]]
-rw-r--r-- 1 phil phil 3352821760 2011-05-25 12:30 test2.qcow2
[[ 26m16s real 1576.363 - user 3.470 - sys 24.340 - 1.76% ]]


removed `test2.qcow2'
`fedora-15-amd64.img' -> `/mnt/vm/fedora-15-amd64.img'
[[ 0m29s real 29.771 - user 0.640 - sys 5.320 - 20.01% ]]
[[ 25m46s real 1546.589 - user 3.040 - sys 18.450 - 1.38% ]]
[[ 0m0s real 0.326 - user 0.000 - sys 0.000 - 0.00% ]]
-rw-r--r-- 1 phil phil 3352821760 2011-05-25 12:56 test2.qcow2
[[ 26m17s real 1577.684 - user 3.680 - sys 24.650 - 1.79% ]]


removed `test2.qcow2'
`fedora-15-amd64.img' -> `/mnt/vm/fedora-15-amd64.img'
[[ 0m29s real 29.133 - user 0.480 - sys 5.450 - 20.35% ]]
[[ 26m1s real 1561.520 - user 3.170 - sys 18.530 - 1.38% ]]
[[ 0m0s real 0.527 - user 0.000 - sys 0.000 - 0.00% ]]
-rw-r--r-- 1 phil phil 3352821760 2011-05-25 13:22 test2.qcow2
[[ 26m32s real 1592.115 - user 3.650 - sys 24.830 - 1.78% ]]
lorentz/phil /home/phil 249>

jefro · 05-25-2011, 03:58 PM

All of the qemu stuff is at the bottom of virtualization apps. Free and works but built in bellards home.

Some of the other vm's have some apps to go to other formats.

Not sure I'd even worry about a raw image versus some other format. They seem to run almost as fast to me.

dyasny · 05-25-2011, 04:28 PM

jefro, you're quite wrong on all accounts. qemu is not Fabrice's baby any longer - lots of good developers working on it nowadays. As for raw vs qcow2 -there's a performance penalty, no doubt, but the extra features are worth it.

Skaperen, I'll try this om my own system tomorrow, I suggest you post your observations to the qemu-devel list, people there might find your workaround interesting

Skaperen · 05-26-2011, 08:19 AM

Quote:

Originally Posted by dyasny

jefro, you're quite wrong on all accounts. qemu is not Fabrice's baby any longer - lots of good developers working on it nowadays. As for raw vs qcow2 -there's a performance penalty, no doubt, but the extra features are worth it.

Skaperen, I'll try this om my own system tomorrow, I suggest you post your observations to the qemu-devel list, people there might find your workaround interesting

Feel free to copy my post there, or reference it (if you are already on the list). I'm not going to sign up for a developer mailing list unless I have an interest in being a developer on that project. IMHO, the fdatasync() calls should be omitted in qemu-img, except for just before closing. I see no need for storage coherency in the middle of running qemu-img since an interrupted run leaves an incomplete result.

I'll probably just stick with raw format, with holes. Both cp and mv replicate zero pages as holes by default (at least in their versions included in Ubuntu 10.10). Rsync has the ability (-S), although it's slightly wrong (last page is never a hole, even when it could be, probably because they didn't want to bother detecting which systems can do that and which cannot). Not sure about some other tools I haven't needed to test, yet.

dyasny · 05-27-2011, 12:27 PM

which qemu version are you running?

Skaperen · 05-27-2011, 04:39 PM

Quote:

Originally Posted by dyasny

which qemu version are you running?

Code:

lorentz/phil /home/phil 1016> qemu --version
QEMU PC emulator version 0.12.5 (qemu-kvm-0.12.5), Copyright (c) 2003-2008 Fabrice Bellard
lorentz/phil /home/phil 1017> dpkg -l | grep qemu
ii  qemu-common                           0.12.5+noroms-0ubuntu7.2                          qemu common functionality (bios, documentation, etc)
ii  qemu-kvm                              0.12.5+noroms-0ubuntu7.2                          Full virtualization on i386 and amd64 hardware
ii  qemu-kvm-extras                       0.12.5+noroms-0ubuntu7.2                          fast processor emulator binaries for non-x86 architectures
ii  qemu-kvm-extras-static                0.12.5+noroms-0ubuntu7.2                          static QEMU user mode emulation binaries
ii  qemu-launcher                         1.7.4-1ubuntu2                                    GTK+ front-end to QEMU computer emulator
ii  qemubuilder                           0.61                                              pbuilder using QEMU as backend
lorentz/phil /home/phil 1018>

dyasny · 05-28-2011, 12:58 AM

apparently, what you describe has been fixed in 0.13, and the current version is .14

Skaperen · 05-31-2011, 12:23 PM

Quote:

Originally Posted by dyasny

apparently, what you describe has been fixed in 0.13, and the current version is .14

Thanks. Hopefully, Ubuntu will have it soon. Otherwise, I'll have to wait until I switch my work desktop over to Slackware, which will likely not be for a few more months.

dyasny · 06-01-2011, 03:42 AM

Quote:

qemu-system-x86-0.14.0-7.fc15.x86_64
gpxe-roms-qemu-1.0.1-4.fc15.noarch
qemu-kvm-0.14.0-7.fc15.x86_64
qemu-img-0.14.0-7.fc15.x86_64
qemu-common-0.14.0-7.fc15.x86_64

This is what I'm running in my Fedora 15. I think the latest *buntu should be at the same patchlevel, or at the very least at the same major version (.14) definitely not 2 major builds behind.

*I know .xx is not major, but it generally is, in qemu terms