Crazy KVM/QEMU slowdown issue I cannot get rid off of

yota86 · 02-07-2015, 04:27 AM

Hello,

I'm having a critical issue with KVM/QEMU which is severely impacting one of my projects and I don't know where to put my hands.

I've set up a brand new physical server configured with a 2 x Intel Xeon E5504 (4c/4t) and 16GB of RAM. I have installed Ubuntu Server 14.04.1 (linux-image 3.13.0-44-generic) along with KVM 2.0.0+dfsg-2ubuntu1.10, QEMU 1.22, libvirt 1.2.2.

I need to set up several VM instances that have to be fired up on demand when I need to analyze big data, with a timeout of 2 minutes, after that I kill the analysis (they should end way before that though). I decided to set up 8 VMs (one per thread) with 1GB of ram assigned each one, and all of them run Windows XP. All the VMs use the qcow2 format.

I installed the first VM and took a snapshot of it (virsh snapshot-create-as), then I replicated the same snapshot on all the other instances.

The first time I run all the VMs everything run smoothly, the VMs fire up and shut down perfectly.

However, after a while I turn them on and off, all the VMs start being slow as hell, like if KVM is not doing anymore hardware emulation and QEMU falls back in software emulation. The consequence is that all the analysis hit the critical timeout and got killed

Even when all the VMs run at the same time the two CPUs are not 100% loaded and the RAM itself is only 56% busy, the swap file is totaly free.

I tried tuning up the I/O scheduler deadline by changing the parameters:

read_expire at 80
write_expire at 1500
fifo batch at 16
front merges at 1
write starved at 6

and I set up the VMs with io=native disk configuration.

This last round of tuning improved the situation, yet it doesn't fix it as I've just raised the amount of time the VMs can run without any issue before reaching the fatal slowdown (which now arrives after several hours).

Could somebody of you guys give me a help in trying to figure out why this behavior is happening?

I don't know anymore where to put my hands on.

Many thanks in advance!

Kind regards

smallpond · 02-08-2015, 07:16 AM

What does running vmstat for a while on the host tell you?

yota86 · 02-08-2015, 01:33 PM

16423004 K total memory
16208756 K used memory
13096368 K active memory
2758504 K inactive memory
214248 K free memory
42400 K buffer memory
7133724 K swap cache
11009016 K total swap
1407108 K used swap
9601908 K free swap
22247892 non-nice user cpu ticks
9116 nice user cpu ticks
12448272 system cpu ticks
188569967 idle cpu ticks
11201184 IO-wait cpu ticks
1677 IRQ cpu ticks
68146 softirq cpu ticks
0 stolen cpu ticks
172340142 pages paged in
392262648 pages paged out
1598289 pages swapped in
17487275 pages swapped out
2478016910 interrupts
2099548744 CPU context switches
1423128247 boot time
3689298 forks

disk- ------------reads------------ ------------writes----------- -----IO------
total merged sectors ms total merged sectors ms cur sec

sda 8037127 1416689 344808388 356302444 9657198 18811257 785300008 774439480 0 69468

iostat -x sda3
Linux 3.13.0-45-generic 08/02/15 _x86_64_ (8 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
9.53 0.00 5.38 4.80 0.00 80.29

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda3 2.89 48.50 23.36 18.54 550.68 1198.84 83.50 2.00 47.73 31.30 68.44 4.97 20.81

yota86 · 02-08-2015, 01:54 PM

iostat -dxk /dev/sda 5

Code:

Linux 3.13.0-45-generic      08/02/15        _x86_64_        (8 CPU)

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               5.09    65.14   27.43   32.84   588.54  1342.31    64.07     3.85   63.88   44.26   80.26   3.93  23.66

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               8.40     6.60   10.60   21.20   238.40  1282.40    95.65     0.43   13.56   16.45   12.11   5.21  16.56

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               2.80     4.80    9.20   15.80   253.60  1180.80   114.75     0.18    7.07    3.74    9.01   4.77  11.92

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               1.60    25.00   32.60   49.20   640.80  1494.40    52.21     1.74   20.93   14.01   25.51   6.60  54.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               6.00    40.40   35.60   87.40   612.00  4059.20    75.95     3.52   28.75   28.81   28.72   8.04  98.88

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda              82.60    32.80   51.20  112.00  1110.40  2823.20    48.21     2.95   18.19   18.36   18.11   5.57  90.96

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               8.60    32.80   76.80   92.40  1233.60  2339.20    42.23     1.94   11.40   10.41   12.23   5.22  88.24

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda              15.40    37.80   51.00   71.60  1516.80  3054.40    74.57     3.79   30.85   32.39   29.74   8.14  99.76

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               8.20    47.40   41.40  121.20   933.60  5107.20    74.30     5.10   31.44   30.82   31.65   6.05  98.40

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda              20.00    78.80   14.40  147.00   224.00  4958.40    64.22     7.09   43.86   36.39   44.59   6.20 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               7.40    49.60    5.60  157.60   109.60  5104.80    63.90     6.25   38.36   22.43   38.93   6.11  99.68

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               7.40    38.80   68.80  119.20   720.00  3260.00    42.34     5.90   31.44   34.22   29.84   4.51  84.72

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda              28.60    26.40   40.80   80.40   964.80  2896.00    63.71     1.00    8.26    5.75    9.53   4.50  54.48

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               7.00   509.40   12.40  741.60   154.40 13492.80    36.20   160.14  194.25   50.13  196.66   1.11  83.92

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda              49.00  6575.40   47.60  337.20   793.60 32258.40   171.79    67.55  211.06   53.29  233.33   2.52  96.88

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda              88.40    69.40   28.00  125.60   628.00  5544.00    80.36     9.47   61.69   46.66   65.04   6.49  99.76

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               3.00    44.20   24.80  168.40   256.80 16989.60   178.53    17.65   91.14   15.10  102.34   5.08  98.24

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda              13.60    37.40  260.20  125.40  2943.20 12859.20    81.96    15.84   41.11   22.52   79.69   2.59 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               9.80 10747.40   39.80  274.40   678.40 53652.00   345.83   117.64  373.34   51.96  419.96   2.93  91.92

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda              17.40    22.20   13.00   64.40   202.40  2319.20    65.16     1.13   14.83   25.42   12.70   5.99  46.40

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               7.80    30.60   13.40   84.20   195.20  2948.80    64.43     1.45   14.75   17.13   14.38   5.75  56.08

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               7.00    24.00   42.80   64.00   473.60  2744.00    60.25     1.24   11.69    7.35   14.60   4.46  47.60

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               7.20    14.20   39.00   51.00   575.20  1975.20    56.68     3.09   34.35   14.46   49.55   4.43  39.84

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda              29.60    30.80   31.40   38.40   664.80  1898.40    73.44     0.80   11.50   12.03   11.06   5.48  38.24

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               7.20    23.00   50.20   80.80  1043.20  3481.60    69.08     1.40   10.53   11.08   10.20   4.53  59.36