LinuxQuestions.org - KVM I/O performance: Raw partitions versus image files

- Linux - Virtualization and Cloud (https://www.linuxquestions.org/questions/linux-virtualization-and-cloud-90/)

- - KVM I/O performance: Raw partitions versus image files (https://www.linuxquestions.org/questions/linux-virtualization-and-cloud-90/kvm-i-o-performance-raw-partitions-versus-image-files-4175447982/)

KVM I/O performance: Raw partitions versus image files

We have a VM framework based on KVM where we slice our drives up into partitions of specific sizes and create our VMs against these partitions. For example, we might have a command such as

Code:

virt-install --connect=qemu:///system \

    --network bridge=br0 \

    --initrd-inject=/ks/ks.cfg \

    --extra-args="ks=file:/ks.cfg text console=tty0 utf8 console=ttyS0,115200" \

    --name=vm-test \

    --disk path=/dev/sda3,bus=virtio \

    --disk path=/dev/sda4,bus=virtio \

    --accelerate \

    --hvm \

    --location=/iso/custom.iso \

    --nographics \

    --noreboot

In this case, our VM would have two virtual drives, /dev/vda (hosting the OS) and /dev/vdb (hosting the VM's core data). These are mapped against physical partitions 3 and 4 of the host's /dev/sda drive. These have been previously created and sized to meet our needs. Additional VMs may share the same physical drive using other partitions.

We were wondering if VMs mapped to physical partitions have any performance advantages over VMs that use pre-allocated image files. For example, instead of paritions sda3 and sda4 above, let's say we did this:

Code:

virt-install --connect=qemu:///system \

...

    --disk path=/vmpool/vm-test-1.img,bus=virtio \

    --disk path=/vmpool/vm-test-2.img,bus=virtio \

...

    --noreboot

where vm-test-1.img and vm-test-2.img would have been previously created image files sized appropriately. I sometimes create VMs in exactly this manner, but I was always under the assumption that using a raw partition would yield better virtual disk performance than using image files. Our VMs can potentially be very I/O intensive, especially with the VM's data drive (/dev/vdb), so we're interested in what is the best approach to take.

The image files I am referring to here would be created using something like dd. I have never experimented with qcow2 image files. Would these be an option?

I am running KVM/libvirt under CentOS 6.3.

I have seen some tests of raw (as in dd) versus qcow2. I'd think that many raw device files could be faster but that leads us to question how one set up the drive mapping and such for the image.

The advantages of qcow 1,2 and three have little to do with speed. They have many other advantages that you may or may not wish to use. A qcow in grow does take a time penalty. Use of snapshots and other features would also but return in it's need or use.

Now as to a dd file or a single partition, I'd guess if both were optimized for the devices, then the test would be mostly equal.

From my simple tests, all hard drive access is counted the same. Only in some compressed formats do the disk numbers increase. That is to say if you had data that could compress well then need to access in on a fast cpu it could exceed the speed of the drive.

Intuitively, I'd think using partitions directly as opposed to image files (even non-sparse) would be faster. A VM accessing its virtual drive would have to go through another translation layer, would it not, if the virtual file system was mapped to a file instead of a raw partition? Or does the KVM drive management bypass the host's file system layer and talk directly to the device when image files are used?

I don't believe it does bypass the hosts drive access principals. In some newer implementations of vm, the host may have access to more real hardware but I don't think that is your case.

The only way to prove would be many tests on vm raw image versus vm mounting of a real partition. I'd guess them to be almost the same. I have never seen actual tests posted. Maybe the qemu or kvm creators or advanced users would know for sure.

That's an interesting post, but unfortunately there are no follow-ups. Plus it doesn't have a comparison with using raw partitions and image files. Ultimately it appears we will have to run our own benchmarks. There's not a lot of information out there...

If you know anyone who's got actual RHEL, they could ask RH.

An excellent suggestion! I just checked with my brother and they run RHEL at their university where he's the Data Center Manager. I'm sure he'll want a favor in return for his help... :-)

Cool; in that case my requested favour is please post any info you get :)

Sadly, after I went into detail what I wanted him to ask Redhat, he said "Wait, we have a educational license. No support."

Alas, I'll need to look elsewhere...

In that case I think you're down to serious googling and/or just trying it...