[SOLVED] High CPU load, top not showing any high CPU process

angel115 · 10-09-2015, 05:32 AM

Hi There,

I've got an issue today with one of our server.
We experience a very high load but top doesn't show wich PID is using all the resources.

Code:

top - 12:27:46 up 47 min,  2 users,  load average: 30.15, 24.36, 21.24
Tasks: 163 total,   1 running, 162 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.2%us,  0.0%sy,  0.0%ni, 33.3%id, 66.5%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   4054488k total,   495592k used,  3558896k free,    15432k buffers
Swap:  2099196k total,        0k used,  2099196k free,   193844k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 7528 root      20   0 19296 1420 1016 R    0  0.0   0:01.45 top
    1 root      20   0 23772 1924 1252 S    0  0.0   0:02.13 init
    2 root      20   0     0    0    0 S    0  0.0   0:00.00 kthreadd
    3 root      20   0     0    0    0 S    0  0.0   0:00.02 ksoftirqd/0
    4 root      20   0     0    0    0 S    0  0.0   0:00.00 kworker/0:0
    5 root      20   0     0    0    0 S    0  0.0   0:00.50 kworker/u:0
    6 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/0
    7 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/1
    8 root      20   0     0    0    0 S    0  0.0   0:00.00 kworker/1:0
    9 root      20   0     0    0    0 S    0  0.0   0:00.02 ksoftirqd/1
   10 root      20   0     0    0    0 S    0  0.0   0:00.18 kworker/0:1
   11 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/2

Any idea where I can look to check which process is eating all the resources?

Best regards,

Emerson · 10-09-2015, 06:05 AM

High load average is really not hogging up your resources, it is merely showing there are lots of processes waiting. You should look at processes in D state.

angel115 · 10-09-2015, 06:09 AM

Thanks, checking...

angel115 · 10-09-2015, 06:13 AM

Ok...

apart of all the apache2 process (which is normal I guess as this a web server) I don't see much things

By the way what is the D state?

Code:

ps -aux | grep "        D    "
Warning: bad ps syntax, perhaps a bogus '-'? See http://procps.sf.net/faq.html
root      1696  0.0  0.0      0     0 ?        D    11:40   0:00 [kjournald]
root     10789  0.0  0.0  84636  3440 ?        D    12:52   0:00 sendmail: MTA: ./t97468dQ016145 from queue
www-data 11340  0.0  0.2 203792  9304 ?        D    12:57   0:00 /usr/sbin/apache2 -k start
root     11932  0.0  0.0  84672  3404 ?        D    13:02   0:00 sendmail: MTA: ./t98BG8AD001049 from queue
www-data 12303  0.0  0.2 203028  8612 ?        D    13:04   0:00 /usr/sbin/apache2 -k start
www-data 12412  0.0  0.2 203720  9256 ?        D    13:05   0:00 /usr/sbin/apache2 -k start
www-data 12448  0.0  0.2 203412  8828 ?        D    13:06   0:00 /usr/sbin/apache2 -k start
www-data 12523  0.0  0.2 203380  8804 ?        D    13:06   0:00 /usr/sbin/apache2 -k start
www-data 12525  0.0  0.2 203156  8760 ?        D    13:06   0:00 /usr/sbin/apache2 -k start
www-data 12597  0.0  0.2 203404  9016 ?        D    13:07   0:00 /usr/sbin/apache2 -k start
www-data 12600  0.0  0.2 207852 10640 ?        D    13:07   0:00 /usr/sbin/apache2 -k start
www-data 12606  0.0  0.2 203692  9152 ?        D    13:07   0:00 /usr/sbin/apache2 -k start
www-data 12609  0.0  0.2 203276  9216 ?        D    13:07   0:00 /usr/sbin/apache2 -k start
www-data 12614  0.0  0.2 203384  8884 ?        D    13:07   0:00 /usr/sbin/apache2 -k start
www-data 12626  0.0  0.2 203300  8920 ?        D    13:07   0:00 /usr/sbin/apache2 -k start
www-data 12628  0.0  0.2 203408  8684 ?        D    13:07   0:00 /usr/sbin/apache2 -k start
www-data 12631  0.0  0.2 203300  9116 ?        D    13:07   0:00 /usr/sbin/apache2 -k start
www-data 12760  0.0  0.2 203356  8356 ?        D    13:08   0:00 /usr/sbin/apache2 -k start
www-data 12842  0.0  0.2 203356  8408 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12843  0.0  0.1 202988  7900 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12844  0.0  0.1 202100  5780 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12845  0.0  0.2 204272  9240 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12846  0.0  0.2 203356  8504 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12848  0.0  0.1 202496  7664 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12850  0.0  0.2 203356  8372 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12851  0.0  0.1 202100  5780 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12854  0.0  0.1 202100  5780 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12855  0.0  0.1 202628  7664 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12856  0.0  0.1 202100  5780 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12858  0.0  0.1 202100  5780 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12861  0.0  0.1 202620  6560 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12865  0.0  0.2 203356  8360 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12866  0.0  0.1 202988  7896 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12867  0.0  0.1 202484  6524 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12868  0.0  0.1 202988  7896 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12869  0.0  0.1 202484  6524 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12870  0.0  0.1 202100  6032 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12871  0.0  0.2 203356  8356 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12876  0.0  0.1 202100  6032 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12877  0.0  0.1 202100  6032 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12878  0.0  0.2 202972  8652 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12879  0.0  0.1 202100  6032 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12880  0.0  0.1 202100  6032 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12881  0.0  0.1 202100  6032 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12882  0.0  0.2 203356  8412 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12883  0.0  0.1 202988  7896 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12884  0.0  0.2 203356  8412 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12885  0.0  0.1 202620  6536 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12886  0.0  0.1 202988  7896 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12887  0.0  0.2 204272  9240 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12889  0.0  0.1 202988  7896 ?        D    13:09   0:00 /usr/sbin/apache2 -k start
www-data 12890  0.0  0.1 202988  7896 ?        D    13:09   0:00 /usr/sbin/apache2 -k start

Emerson · 10-09-2015, 06:19 AM

From man top:

Code:

20. S  --  Process Status
           The status of the task which can be one of:
               D = uninterruptible sleep
               R = running
               S = sleeping
               T = stopped by job control signal
               t = stopped by debugger during trace
               Z = zombie

angel115 · 10-09-2015, 07:01 AM

Ok, thanks.

And do you know what could possibly create this issue?
As nothing was touch on the server.

Does HDD latency could create this type of issue?

Thanks for your insight

szboardstretcher · 10-09-2015, 08:33 AM

This has been discussed: http://www.linuxquestions.org/questi...ry-low-315210/

chrism01 · 10-11-2015, 08:31 PM

I'd want to know what those apache processes are actually trying to do - DB access by any chance ? Maybe nfs mount access?

angel115 · 10-12-2015, 12:58 AM

Hi All,

Problem solved.

This was a latency issue on our SAN.
After migrating the VM on an other SAN storage the problem is now solved.

I've notice that with the high "wa" on the top command which was on average at 80 +.

Now the "wa" is beween 0 and 2%.

Thank you all for your precious help.

Angel.