Extremaly high server load 2-3 times per month

lpa · 07-16-2009, 05:03 AM

Hello,

I have the following problem. Usually the average load of my server (CentOS) is about 0.5 - 2. Every month there are 1-2 black days when happends the following:

PHP Code:



- SIM Log: 
[07/16/09 10:40:01]: MYSQL service is online. 
[07/16/09 10:45:01]: LOAD 0.58 (status good) 
[07/16/09 10:45:01]: HTTP service is online. 
[07/16/09 10:45:01]: MYSQL service is online. 
[07/16/09 10:50:01]: LOAD 1.50 (status good) 
[07/16/09 10:50:01]: HTTP service is online. 
[07/16/09 10:50:01]: MYSQL service is online. 
[07/16/09 10:55:01]: LOAD 1.12 (status good) 
[07/16/09 10:55:01]: HTTP service is online. 
[07/16/09 10:55:01]: MYSQL service is online. 
[07/16/09 11:00:01]: LOAD 1.06 (status good) 
[07/16/09 11:00:01]: HTTP service is online. 
[07/16/09 11:00:01]: MYSQL service is online. 
[07/16/09 11:05:22]: LOAD 76.95 (status critical) 
[07/16/09 11:05:22]: load status critical, non-essential services going down.

As you can see, in 5 minutes server load grows from 1 to 77. When it happends, the problem usually repeats 2-3 times right after the stopped services are restarted. This happends momentally and I can not even check the top because server is almost frozen.

It happends only 1-2 times per month. I have only 1 website on this server and there are no scripts which are accessed only 1-2 times per month, so this couldn't be caused by bad scripts. I have one more server (Debian) with the same scripts and there are no problems like this. There are no other users on my server.

There are no spammers who are sending spam from my server.

I don't have any cron jobs that run only 1-2 times per month and may cause such a load.

I need any advice how to fix this problem and how to find out what causes it.

Thank you.

syg00 · 07-16-2009, 05:25 AM

How often does this come up ... ?
loadavg is (runq+uninterruptible sleep). Generally that's disk I/O wait.
Try this when you have the "problem":

Code:

top -b -n 1 | awk '{if (NR <=7) print; else if ($8 == "D") {print; count++} } END {print "Total status D: "count}'

rylan76 · 07-16-2009, 05:28 AM

Not sure if this will help, but do you have AWStats or something similar running on that machine?

Alternatively, it should be simple to keep record of what IP addresses visit your site.

I. e. if you have a "load hit" taking place, wait for it to pass, then go and check AWStats or your IP log list.

I once had much the same problem with one of my sites - except its bandwidth use was running away completely. I used a log of visiting IP adresses and correlated that with the hourly bandwidth use. I quickly discovered the culprit IP - which turned out to be a University in another city that was excessively caching the site on their local intranet for their users.

So you should be able to do much the same, only you won't correlate against bandwidth use over time but against load over time. I'm betting you have some EXTERNAL server or script hitting your server to update a local cache on its side, or something similar. The fact that you state this happens only once or twice a month might point to something automatic, somewhere, really hitting your server hard.

I'm assuming the exact times vary? I. e. it is always at different times?

lpa · 07-16-2009, 07:13 AM

Hello,

Thank you for your replies.

No, I don't have AWStats installed or something similar.

I have this server since September 2008, but as I remember, the problems started in January. Unfortunately, I have deleted some warning messages from SIM, but here are 4 which I found in my mailbox.

Monday, January 12, 11:00
Monday, June 22, 14:00
Monday, July 13, 14:40
Thursday, July 16, 11:05

As you can see, 3 times of 4 the problem was on Monday. The times are different, but it always happends in time range from 10 am till 4 pm. Usually it happends not often than 2 times per month, but in July it happened already 2 times.

When the problem started, SIM reported about server loads like 20-25. Then it grew to ~30-35, but today the load was 77!

I know that some crawlers may cause high server loads, but this is a dating website with not so many information which is accessible without logging in. Crawlers can access only the startpage which shows the last discussions, last profiles, etc. At the same time lot of profiles are "open" and accessable by everyone without logging in. I have noticed that Google is indexing images from user profiles, but could it cause server loads like 70?