Need Help with High Load Average & High Sleeping Task

js9028 · 07-08-2019, 12:27 PM

My server load average is high, as well as the high sleeping task. When I use the top function, this is what appeared:

Code:

top - 00:19:57 up 884 days,  6:58,  1 user,  load average: 8.66, 12.39, 16.79
Tasks: 659 total,   2 running, 657 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.1 us,  0.6 sy,  0.0 ni, 74.5 id, 24.9 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  7904908 total,  1783236 free,  1534192 used,  4587480 buff/cache
KiB Swap:  8142844 total,  8110288 free,    32556 used.  5607916 avail Mem

Then, I tried to see what those sleeping processes are, and it's a very long list. Here's a sample:

[kthreadd]
[ksoftirqd/0]
[kworker/0:0H]
[migration/0]
[rcu_bh]
[rcuob/0]
[rcuob/1]
[rcuob/2]
[rcuob/3]
[rcuob/4]
[rcuob/5]
[rcuob/6]
[rcuob/7]
[rcuob/8]
[rcuob/9]
[rcuob/10]
[rcuob/11]
[rcuob/12]
[rcuob/13]
[rcuob/14]
[rcuob/15]
[rcuob/16]
[rcuob/17]
[rcuob/18]
[rcuob/19]
[rcuob/20]
[rcuob/21]
[rcuob/22]
[rcuob/23]
[rcuob/24]
[rcuob/25]
[rcuob/26]
[rcuob/27]
[rcuob/28]
[rcuob/29]
[rcuob/30]
[rcuob/31]
[rcuob/32]
[rcuob/33]
[rcuob/34]
[rcuob/35]
[rcuob/36]
[rcuob/37]
[rcuob/38]
[rcuob/39]
php-fpm: pool www
php-fpm: pool www
php-fpm: pool www
[kworker/1:1]
php-fpm: pool www
php-fpm: pool www
php-fpm: pool www
php-fpm: pool www
php-fpm: pool www
php-fpm: pool www
php-fpm: pool www
php-fpm: pool www
php-fpm: pool www
php-fpm: pool www
php-fpm: pool www
php-fpm: pool www
php-fpm: pool www
top
[kworker/3:1]
[kworker/0:1]
[kworker/u480:2]
[kworker/3:3]
sshd: andi [priv]
sshd: andi@notty
/usr/libexec/openssh/sftp-server
[kworker/2:0]
[kworker/1:0]
php-fpm: pool www
php-fpm: pool www
php-fpm: pool www
php-fpm: pool www
php-fpm: pool www
php-fpm: pool www
php-fpm: pool www
php-fpm: pool www
[kworker/2:3]
[kworker/3:0]
[kworker/3:2]
sshd: andi [priv]
/usr/libexec/openssh/sftp-server
[kworker/1:2]
local -t unix
bounce -z -t unix -u
php-fpm: pool www
php-fpm: pool www
php-fpm: pool www
php-fpm: pool www
trivial-rewrite -n rewrite -t unix -u
smtpd -n smtp -t inet -u -o stress= -s 3
smtpd -n smtp -t inet -u -o stress= -s 3
grep --color=auto ^[SD]
cut -b 3-
/usr/sbin/lvmetad -f

--
What do I do? Should I terminate the sleeping process? What's the best way? Because basically, my web application cannot be accessed and it's very slow. I should add that the load average can get to 20, 30, etc.

Background info: My server is a 4core server, linux CentOS, 8GB RAM.
--

Please help! Thank you.

syg00 · 07-08-2019, 10:45 PM

All non-looping tasks spend very close to all the time sleeping. That's why multi-tasking works at all. So the high sleeping count is "normal".
Tasks with square brackets are usually kernel threads, so ignore them almost always. That leaves php - which I know nothing about.

This is a better indicator of what's wrong

Code:

%Cpu(s):  0.1 us,  0.6 sy,  0.0 ni, 74.5 id, 24.9 wa,  0.0 hi,  0.0 si,  0.0 st

You're not using much swap so I'd guess all those php tasks are queuing up trying to write (or maybe read) from the hard disk.
Most likely an application design problem.

MadeInGermany · 07-20-2019, 12:48 AM

Waiting or blocked proccesses wait for output to drain; many of them indicate a bottleneck in the I/O: disk or network.
Look for proccesses in D state

Code:

ps -elf grep -w D
ps -eo state,pid,args | awk '$1~/D/'

The second command is more precise. Run the command several times.
The D processes want to write more than the I/O device can handle.

pan64 · 07-20-2019, 08:55 AM

Looks like a lot of I/O waiting time, otherwise OK. The load and the number of sleeping processes means nothing, that can be still ok.
If it was a web application I would vote on network: can you check if the network connection is ok?
From the other hand you can check /var/log for any messages related to hardware (any kind of defect, including HDD, network or anything else).