Hey, so I have a few servers on my XenServer, and I don't know how to track down the problem I'm seeing:

Every so often it seems the HyperVisor has a lot of ioWait:


12:20:01 PM all 0.05 0.00 0.05 0.74 0.23 98.93
12:30:01 PM all 0.26 0.00 11.68 7.63 0.80 79.63
12:40:01 PM all 0.07 0.00 1.84 1.52 0.55 96.01
12:50:01 PM all 0.04 0.00 0.06 0.90 0.44 98.56
01:00:01 PM all 0.05 0.00 0.15 1.22 0.27 98.30
01:10:02 PM all 0.10 0.00 0.16 3.10 0.32 96.31
01:20:01 PM all 0.05 0.00 0.06 1.25 0.29 98.35
01:30:01 PM all 0.28 0.00 10.79 13.55 0.79 74.58


This then impacts the Vm's, mainly one main VM, which is probably the cause, but I'm not sure where to start finding the problem.

The main VM on the server has the following:


12:25:02 all 2.88 0.00 5.40 10.25 0.00 81.47
12:35:01 all 2.98 0.00 7.35 15.35 0.00 74.32
12:45:01 all 3.14 0.00 0.59 0.85 0.00 95.41
12:55:01 all 3.46 0.00 0.65 0.72 0.00 95.17
13:05:01 all 3.32 0.00 0.58 1.45 0.00 94.65
13:15:01 all 3.56 0.00 0.68 0.84 0.00 94.92
13:25:01 all 3.14 0.00 5.17 13.10 0.00 78.59
13:35:01 all 4.02 0.00 7.53 14.54 0.00 73.92


I hope you guys can help me, or at least point me towards what else to investigate!

iostat output would be a better place to look - at least then you'd have an idea of who is generated the I/O load.
Depending on your sysstat level (and kernel options) it also has pidstat, but that is only interactive.

collectl allows you to drill down to the process level after the event. You might want to consider that as an option (in future).

If you delete every tool that shows IOWAIT or simply ignore it, you'd be doing yourself a big favor. That's one of the most useless counters there is because it does NOT indicate any real CPU load and most people do not realize this. In fact, there might be a few tools floating around that include it when they report total CPU load and they shouldn't.
-mark