Centos KVM Tuning questions

terrytibbs · 05-28-2012, 11:28 AM

Hi,

I have a Centos 6.2 host running with one KVM Centos 5.8 guest.

The guest is running as a web server (PHP). The existing web servers are all running on physical hardware, I am seeing a massive increase in CPU load on the guest when compared to the other physical hosts.

I am expecting an overhead due to virtualisation but think I have a bottle neck somewhere.

I have the below setup:

* Using a block device for the guests root disk
* Using virtio drivers for Network and IO
* Pinned Virtual CPU's to the guest
* set deadline scheduler for the host and guest
* disk cache is set to "none"

My top look like this:

Code:

top - 09:21:21 up 11 min,  2 users,  load average: 27.17, 17.06, 7.84
Tasks: 347 total,  11 running, 336 sleeping,   0 stopped,   0 zombie
Cpu(s): 34.8%us, 11.2%sy,  0.0%ni, 47.4%id,  0.6%wa,  1.1%hi,  4.9%si,  0.0%st
Mem:  10237520k total,  4304288k used,  5933232k free,    16912k buffers
Swap: 16386292k total,        0k used, 16386292k free,   294168k cached

You can see my CPU's are being hammered but I'm not sure whats causing this. The load on the physical servers at this time was around 2.5. I would expect a trade off but not this much!

Any ideas?

custangro · 05-28-2012, 12:36 PM

Quote:

Originally Posted by terrytibbs

Hi,

I have a Centos 6.2 host running with one KVM Centos 5.8 guest.

The guest is running as a web server (PHP). The existing web servers are all running on physical hardware, I am seeing a massive increase in CPU load on the guest when compared to the other physical hosts.

I am expecting an overhead due to virtualisation but think I have a bottle neck somewhere.

I have the below setup:

* Using a block device for the guests root disk
* Using virtio drivers for Network and IO
* Pinned Virtual CPU's to the guest
* set deadline scheduler for the host and guest
* disk cache is set to "none"

My top look like this:

Code:

top - 09:21:21 up 11 min,  2 users,  load average: 27.17, 17.06, 7.84
Tasks: 347 total,  11 running, 336 sleeping,   0 stopped,   0 zombie
Cpu(s): 34.8%us, 11.2%sy,  0.0%ni, 47.4%id,  0.6%wa,  1.1%hi,  4.9%si,  0.0%st
Mem:  10237520k total,  4304288k used,  5933232k free,    16912k buffers
Swap: 16386292k total,        0k used, 16386292k free,   294168k cached

You can see my CPU's are being hammered but I'm not sure whats causing this. The load on the physical servers at this time was around 2.5. I would expect a trade off but not this much!

Any ideas?

It would help to know a little more about your setup...

* When running a "top" can you press "1" (the number one) to see all CPUs? Maybe one is getting hammered while the others stay idle...
* Where are you storing the VD? On the physical machine or an NFS mount or iSCSI?
* How is the KVM server installed? Did you install "full GUI" or did you install "Minimal Server"? (I've had problems where the "Fingerprint reader" was crashing...on a Server! A very

move on my part...)

Also, are you sharing the same network interface as the KVM host? Or is it a dedicated NIC for virtual guests?

-C

terrytibbs · 05-30-2012, 08:24 AM

The disk for this VM is a block device, using the virtio driver, I have a dedicated bridge br1 for this VM. I ran the web server on the physical blade last night just to make sure there was not a problem outside of KVM. I got the expected results from physical hardware.

I'm back using KVM, however.....

I had 16 virtual CPU's available to the host. I read that over committing CPU resource can impact performance, so after reducing Current Allocation to 8 I am seeing the load that resembles the physical servers. I still have some tuning to do, as load is running at 4 in the VM where load is 2 on the physical hardware. But this is closer to the expected results.

Thanks very much for taking the time to reply.

custangro · 05-30-2012, 10:15 AM

I'm glad you figured out your issue, and thank you for posting your findings