High load average all of a sudden

Spuddy · 07-12-2010, 07:14 AM

Hello,

I am new here and generally new to Linux. We have a Debian server running apache2. Our load average used to be below 1.0, however this week it is now at anywhere between 5-10.

It is causing real big problems. It is our front end client facing HTTP server. We have coded PHP to turn away requests if the load average is above 2.0. Does anyone have any advice? If I disable the apache2 service, the load average drops back down to an acceptable level, re-enabling it rockets the load average back up.

I'm not extremely well versed in linux so be gentle!

Thanks,

Code:

top - 12:25:42 up  1:24,  2 users,  load average: 6.23, 6.38, 5.99
Tasks: 132 total,   2 running, 129 sleeping,   0 stopped,   1 zombie
Cpu(s): 48.9%us,  3.7%sy,  0.0%ni, 47.2%id,  0.0%wa,  0.1%hi,  0.2%si,  0.0%st
Mem:   4045104k total,   746116k used,  3298988k free,    21468k buffers
Swap:  1951888k total,        0k used,  1951888k free,   589548k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 4762 www-data  16   0  117m  10m 3528 S    8  0.3   0:01.68 apache2
 5845 www-data  16   0  117m 9.9m 3372 S    8  0.3   0:00.60 apache2
30025 www-data  15   0  117m  12m 5316 S    8  0.3   0:09.91 apache2
 1758 www-data  15   0  117m  10m 3900 S    8  0.3   0:04.70 apache2
 3916 www-data  15   0  117m  10m 3600 S    8  0.3   0:02.85 apache2
  903 www-data  15   0  117m  10m 3852 S    8  0.3   0:02.53 apache2
27227 www-data  16   0  117m  11m 4796 S    7  0.3   0:11.85 apache2
27308 www-data  15   0  117m  10m 4024 S    7  0.3   0:12.12 apache2
 8208 www-data  15   0  117m  10m 4216 S    7  0.3   0:26.56 apache2
 4014 www-data  15   0  117m  10m 3776 S    7  0.3   0:02.52 apache2
 4197 www-data  15   0     0    0    0 Z    7  0.0   0:02.35 apache2 <defunct>
 5937 www-data  15   0  117m 9.9m 3344 S    7  0.2   0:00.41 apache2
22952 www-data  15   0  117m  12m 5364 S    7  0.3   0:15.06 apache2
 1593 www-data  15   0  116m  10m 3732 S    7  0.3   0:04.34 apache2
 3494 www-data  15   0  117m  10m 3760 S    7  0.3   0:02.88 apache2
28638 www-data  15   0  117m  12m 5304 S    6  0.3   0:10.16 apache2
30262 www-data  15   0  117m  10m 4152 S    6  0.3   0:07.92 apache2
 5843 www-data  15   0  116m 9.9m 3352 S    6  0.2   0:00.59 apache2
 1148 www-data  15   0  116m  10m 3912 S    6  0.3   0:05.84 apache2
 3416 www-data  16   0  116m  10m 3604 S    6  0.3   0:03.35 apache2
 5935 www-data  15   0  117m 9.9m 3352 R    6  0.2   0:00.40 apache2
 5950 www-data  15   0  116m 9.9m 3340 S    6  0.2   0:00.41 apache2
30622 www-data  15   0  117m  12m 5296 S    6  0.3   0:08.80 apache2
28334 www-data  16   0  117m  10m 4120 S    5  0.3   0:09.82 apache2
 3641 www-data  16   0  116m  10m 3564 S    5  0.3   0:02.70 apache2
14353 www-data  16   0  117m  10m 4172 S    5  0.3   0:24.37 apache2
27226 www-data  15   0  117m  10m 4212 S    5  0.3   0:12.14 apache2
25541 www-data  15   0  117m  12m 5504 S    4  0.3   0:14.05 apache2
 3495 www-data  15   0  117m  10m 3792 S    3  0.3   0:03.01 apache2
 5578 www-data  15   0  117m 9.9m 3392 S    2  0.3   0:00.72 apache2
 5764 www-data  15   0  116m 9.9m 3380 S    2  0.3   0:00.52 apache2
    1 root      15   0  6124  692  568 S    0  0.0   0:02.01 init
    2 root      RT   0     0    0    0 S    0  0.0   0:00.19 migration/0
    3 root      34  19     0    0    0 S    0  0.0   0:00.00 ksoftirqd/0
    4 root      RT   0     0    0    0 S    0  0.0   0:00.00 watchdog/0
    5 root      RT   0     0    0    0 S    0  0.0   0:00.09 migration/1
    6 root      34  19     0    0    0 S    0  0.0   0:00.00 ksoftirqd/1
    7 root      RT   0     0    0    0 S    0  0.0   0:00.00 watchdog/1
    8 root      RT   0     0    0    0 S    0  0.0   0:00.06 migration/2
    9 root      34  19     0    0    0 S    0  0.0   0:00.00 ksoftirqd/2
   10 root      RT   0     0    0    0 S    0  0.0   0:00.00 watchdog/2
   11 root      RT   0     0    0    0 S    0  0.0   0:00.12 migration/3
   12 root      34  19     0    0    0 S    0  0.0   0:00.00 ksoftirqd/3
   13 root      RT   0     0    0    0 S    0  0.0   0:00.00 watchdog/3
   14 root      10  -5     0    0    0 S    0  0.0   0:00.02 events/0
   15 root      10  -5     0    0    0 S    0  0.0   0:00.01 events/1
   16 root      10  -5     0    0    0 S    0  0.0   0:00.06 events/2
   17 root      10  -5     0    0    0 S    0  0.0   0:00.00 events/3
   18 root      10  -5     0    0    0 S    0  0.0   0:00.01 khelper
   19 root      11  -5     0    0    0 S    0  0.0   0:00.00 kthread
   26 root      10  -5     0    0    0 S    0  0.0   0:00.02 kblockd/0

syg00 · 07-12-2010, 07:44 AM

Quote:

Originally Posted by Spuddy

We have coded PHP to turn away requests if the load average is above 2.0.

And who decided that ?. And on what basis. Looks like (at least) a 4 (hyper-)thread machine. Possibly real cores.
Is the (so called) "high" loadavg affecting service to your clients - if not, why are you turning them away ?.

Spuddy · 07-12-2010, 07:51 AM

Quote:

Originally Posted by syg00

And who decided that ?. And on what basis. Looks like (at least) a 4 (hyper-)thread machine. Possibly real cores.
Is the (so called) "high" loadavg affecting service to your clients - if not, why are you turning them away ?.

Thank you for you response, it is much apprieciated.

The server is a quad Xeon E5410 server with 4GB RAM. The decision to turn away clients at 2.0 load average was made before my time. Like I say, I'm new to linux and am learning.

Whilst I understand what you are saying, it still doesn't explain why the server had a low load average last week, but this week it's so high!?

What should we set the max load average to before turning clients away?

Thanks,

syg00 · 07-12-2010, 08:06 AM

How high a tree would you like to be lynched from ?.
Bad metric to use IMHO - people who do tend to have came from classic Unix (not Linux) backgrounds. Loadavg has a different meaning here. Run this (when the loadavg is high) and post the output

Code:

top -b -n 1 | awk '{if (NR <=7) print; else if ($8 == "D") {print; count++} } END {print "Total status D: "count}'

Spuddy · 07-12-2010, 08:11 AM

Output doesn't look great?

Code:

++} } END {print "Total status D: "count}'int; else if ($8 == "D") {print; count
top - 14:09:53 up  3:08,  2 users,  load average: 4.88, 5.38, 5.17
Tasks: 150 total,   4 running, 145 sleeping,   0 stopped,   1 zombie
Cpu(s): 58.3%us,  4.4%sy,  0.0%ni, 36.9%id,  0.1%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:   4045104k total,  1449748k used,  2595356k free,    43468k buffers
Swap:  1951888k total,        0k used,  1951888k free,  1240276k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
Total status D:

syg00 · 07-12-2010, 08:29 AM

One of the consequences of using sampled data. I'm sure innumerable PhD's have been earned analysing such conundrums.
To some extent it proves my point. Even with the Unix definition, you need (consistently) more than 4 runnable tasks to see any delay for CPU resources. Which is usually what "loadavg" is seen as. With your kit you can always run 4 tasks concurrently. If you did, you'd have no (CPU) delay, but a loadavg of at least 4.

Why don't you run that command in a loop with a sleep and see what you get over time. In the meantime, if you think the number you are currently using may have been set based on a single core machine, just bump it to 8. That may be closer to reality.

maybe.

salasi · 07-12-2010, 05:08 PM

Quote:

Originally Posted by Spuddy

It is causing real big problems. It is our front end client facing HTTP server. We have coded PHP to turn away requests if the load average is above 2.0.

Hmm, does raise the question whether you would be having a problem if you weren't taking this measure, but

Quote:

If I disable the apache2 service, the load average drops back down to an acceptable level, re-enabling it rockets the load average back up.

Well, you are running quite a few instances of Apache and each one takes a few percent of CPU and that adds up. So, that's what you'd expect.

Quote:

Our load average used to be below 1.0, however this week it is now at anywhere between 5-10.

Something has changed. It might be a software update that has some extra 'feature', it might be something to do with hardware, it might the server facing a higher level of load and thus starting more instances of Apache. It might even be an attack attempt (doesn't slow loris do something vaguely like this?), in which case maybe you policy with dropping requests above a certain load level is quite good (maybe you could allow a higher load number, possibly).

But, if you can get some kind of clue as to what has changed, you'll be quite a bit further on.

markseger · 07-15-2010, 06:12 AM

try monitoring with collectl. it can be your friend! see my reply to "High Load Average, Low CPU, Low IO Wait" posted by "ordaolmayanadam" and which is a couple of dozen posts after (above) this one and I responded with much to text to retype and didn't want to start parallel threads. lots of options to get the data you're looking for...
-mark

Spuddy · 07-15-2010, 06:37 AM

hello,

Thanks for all your help. We managed to resolve this in the end by deleting both the access.log and ssl_access.log files. Neither was particularly big (255mb) but for some reason this dropped the load from 10ish, to 0.1#.

THanks again,