mpstat output interpret and process info

thllgo · 05-19-2011, 03:38 PM

Hello,

I've got a system that is maxing out CPU. Running "mpstat -P ALL 3 5" shows the following.

Quote:

08:30:21 PM CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s
08:30:24 PM all 100.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1001.00
08:30:24 PM 0 100.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1001.00
08:30:24 PM 1 99.67 0.00 0.00 0.00 0.00 0.00 0.00 0.33 0.00
08:30:24 PM 2 100.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
08:30:24 PM 3 100.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

All 5 groups show the same. My first question is why would CPU0 be the only one with intr/s and the others do not?
Info. The OS is RHEL 5.4 running as a VM on ESXi 4.1. Memory doesn't appear to be an issue, the system has 8 GB and its only using about 1.5 GB.

Second question, I'm positive the process that is the problem is the tomcat process. Does anyone know a good way to see whats happening with a specific process?

SteveK1979 · 06-15-2011, 05:04 PM

Hi,

Quote:

Second question, I'm positive the process that is the problem is the tomcat process. Does anyone know a good way to see whats happening with a specific process?

strace is your friend! It should show you what the tomcat process is up to at a fairly low level.

As to the interrupts only being handled by one cpu, I would have to guess this is a vaguery of the fact it's a vm. I checked a VM that has quad cpus allocated and I see exactly the same behavoir.

Cheers,
Steve

thllgo · 06-16-2011, 09:29 AM

Thank you,
strace is my new little friend.

I'm going to have to check with VMware. I'm wondering if the behavior with the lack of distribution between CPUs has a fix.

SteveK1979 · 06-16-2011, 03:09 PM

Hi,

What makes you think this is actually a problem that needs fixing? There are quite a few things that can influence this too. It can be influenced by HyperThreading as only one of the 'threads' can be interrupted since there is only one cpu in reality.

I'm quite interested in your answer - we also have a large number of linux VMs on VMware.

Cheers,
Steve

thllgo · 06-16-2011, 04:10 PM

The only difference between this and our other system is this system is running a newer version of our software that is wrapped around tomcat. Its not tomcats fault its something we are doing. We have a base VM that we copy and test our SW on. At the end of the test we fix our SW, destroy the VM and install the new version of the SW on a new VM. This way we have a clean system for each new version of the SW. This time however the system idle never drops bellow about 85%, frequently reaching 100%. Without feeding the system any data it is consistently using way too much CPU. Past tests show an idle system only around 5%.

SteveK1979 · 07-07-2011, 03:37 PM

Hi,

Did you ever get a response from VMware for this? Out of curiosity is irqbalence running on these systems?

Cheers,
Steve

thllgo · 07-12-2011, 09:51 AM

We did find the issue. One of the developers while fixing one problem created another that put one section of the app. into a CPU sucking loop.

, but we found it and fixed it.

I was sure the issue was our app.