Question about %CPU from top command

smecherel · 05-22-2014, 03:26 AM

Hi,
I have problem understanding the below output from top and after spending some time searching on google, i decide to ask for help.
__________________________________________________________________________
cosmin@cosmin1:~$ top
top - 11:21:46 up 22 min, 1 user, load average: 0.71, 0.66, 0.51
Tasks: 118 total, 1 running, 117 sleeping, 0 stopped, 0 zombie
Cpu0 : 10.2%us, 23.3%sy, 0.0%ni, 66.5%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 7.5%us, 26.4%sy, 0.0%ni, 66.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 4049884k total, 586612k used, 3463272k free, 43436k buffers
Swap: 4192252k total, 0k used, 4192252k free, 299656k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1156 bind 20 0 235m 18m 3024 S 83 0.5 18:31.87 named
1243 root 20 0 151m 18m 9744 S 0 0.5 0:01.36 Xorg
___________________________________________________________________________
From what i know, 66.5%id (in case of CPU0) means that in 66.5% of the time the CPU is idle, doing nothing. The rest of the time, 33.5% (10.2%us + 23.3%sy) is doing user and system tasks.
What is the meaning of 83% CPU usage for named process? Is it 83% of the 33.5%?
Or is it 83% from total CPU? If this is the case how come both CPU are idle for about 66%?

hairysocks · 05-22-2014, 04:52 AM

I think it is a problem with the way top works with multiple-cpu machines.

Look at the output from one of my servers:

Code:

top - 10:40:55 up 1519 days, 23:23,  1 user,  load average: 1.54, 1.32, 1.30
Tasks: 122 total,   1 running, 120 sleeping,   0 stopped,   1 zombie
Cpu0  : 32.0% us,  3.0% sy,  0.0% ni, 65.0% id,  0.0% wa,  0.0% hi,  0.0% si
Cpu1  : 21.3% us,  2.3% sy,  0.0% ni, 76.1% id,  0.3% wa,  0.0% hi,  0.0% si
Cpu2  : 24.0% us,  2.7% sy,  0.0% ni, 73.3% id,  0.0% wa,  0.0% hi,  0.0% si
Cpu3  : 20.9% us,  2.7% sy,  0.0% ni, 76.4% id,  0.0% wa,  0.0% hi,  0.0% si
Cpu4  : 20.6% us,  2.0% sy,  0.0% ni, 77.4% id,  0.0% wa,  0.0% hi,  0.0% si
Cpu5  : 23.7% us,  2.3% sy,  0.0% ni, 74.0% id,  0.0% wa,  0.0% hi,  0.0% si
Cpu6  : 26.3% us,  2.0% sy,  0.0% ni, 71.7% id,  0.0% wa,  0.0% hi,  0.0% si
Cpu7  : 20.7% us,  2.0% sy,  0.0% ni, 77.3% id,  0.0% wa,  0.0% hi,  0.0% si
Mem:   4147716k total,  3392900k used,   754816k free,    46088k buffers
Swap:  2031608k total,    13772k used,  2017836k free,  2145252k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
29785 mysql     15   0  202m  52m 2444 S 181.1  1.3  24581:36 mysqld
15040 tomcat    17   0 2014m 811m  25m S 23.3 20.0  33:31.71 java
10196 root      17   0  5924  588  308 S  0.3  0.0 733:37.56 bash
    1 root      16   0  3376  116   68 S  0.0  0.0  16:34.77 init

it shows 181.1% activity for mysql - which is quite close to the value 189.5 that you get adding up the "us" % values for all CPUs. Of course, we all know you can't add percentages like that!

johnsfine · 05-22-2014, 05:37 AM

Quote:

Originally Posted by smecherel

What is the meaning of 83% CPU usage for named process? Is it 83% of the 33.5%?

No.

Quote:

Or is it 83% from total CPU?

No. It is the equivalent of 83% of one core. The OS switched that process between two cores. So 83% might mean 40% of one core plus 43% of another.

Quote:

If this is the case how come both CPU are idle for about 66%?

You really have only two CPUs ?

Sometimes the values you see in top are wrong due to sampling glitches. With two CPUs idle for the 132.5% of one core you showed, the total of all processes should be almost all of the remaining 67.5%, not more.

Quote:

Originally Posted by hairysocks

it shows 181.1% activity for mysql - which is quite close to the value 189.5 that you get adding up the "us" % values for all CPUs. Of course, we all know you can't add percentages like that!

I don't know "you can't add percentages like that" and the people who wrote top obviously didn't know it either. Because adding percentages like that is exactly what they are doing. The sy percentages are also included in the sum of percentages for the process that was selected at the time the sy CPU time was consumed. Your 181% is less than the sum of us and sy, because another process used a significant amount.

sundialsvcs · 05-22-2014, 06:23 AM

As I understand it, these figures are basically trying to be calculations of wall-time. And these figures are "apples and oranges."

For a processor (or core), the calculation shows how much wall-time that processor spent in various states during the sample period.
Meanwhile, for a process, the calculation shows how much wall-time the process spent running, or in various kinds of involuntary wait-states.

But what top really can't show you is how the running workload happened to distribute itself among the various processors/cores over that time period. Especially when the overall workload is light, this can be pretty-nigh impossible to rationalize. Sometimes, the numbers are "simply, obviously-nonsense," like 181%.

There are at least two useful things that you can take-away from top, however:

"Why are we waiting?" What is a process waiting on? (When it would prefer not to be waiting at all ...) Virtual-memory is a particular thing to be watching for.
"The 10,000-pound elephants in the room." Particular processes that are consuming a disproportionate amount of any resource, such that they delay others. (The "CPU" resource can generally be regarded as free-of-charge.)

The figures are generally best taken as "relative to one another, not absolute." The exact values are somewhat arbitrary, but the relationship among them over time can be useful.

johnsfine · 05-22-2014, 07:41 AM

Quote:

Originally Posted by sundialsvcs

Sometimes, the numbers are "simply, obviously-nonsense," like 181%.

It isn't nonsense. It is very well defined. It is % that process used of each processor summed over all processors. For example, it might have used 81% of one processor plus 60% of another, plus 40% of a third.

syg00 · 05-22-2014, 09:26 AM

The task CPU% is non-normalised - a multi-threaded process may sum to (considerably) more than 100%. Each thread is limited by the laws of physics to 100% - i.e. equivalent to all of 1 "core".

If you are interested in "dynamic" usage numbers, you must ignore the first iteration of the top output - there is no "previous" entry to determine the delta, so uses the usage since boot for the summary, or process start for the task data. It is similar to the numbers expected from "ps".
Run "top -b -n 2 > somefile" and see the difference.

smecherel · 05-23-2014, 01:55 AM

Thanks for your quick response. Now it is more clear to me how top works. In the mean time, i did find what named was doing to keep 83% CPU and solved it.