Latest LQ Deal: Linux Power User Bundle
Go Back > Forums > Linux Forums > Linux - Newbie
User Name
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!


  Search this Thread
Old 07-16-2009, 05:03 AM   #1
LQ Newbie
Registered: Jul 2009
Posts: 6

Rep: Reputation: 0
Extremaly high server load 2-3 times per month


I have the following problem. Usually the average load of my server (CentOS) is about 0.5 - 2. Every month there are 1-2 black days when happends the following:

PHP Code:
SIM Log:
07/16/09 10:40:01]: MYSQL service is online.
07/16/09 10:45:01]: LOAD 0.58 (status good)
07/16/09 10:45:01]: HTTP service is online.
07/16/09 10:45:01]: MYSQL service is online.
07/16/09 10:50:01]: LOAD 1.50 (status good)
07/16/09 10:50:01]: HTTP service is online.
07/16/09 10:50:01]: MYSQL service is online.
07/16/09 10:55:01]: LOAD 1.12 (status good)
07/16/09 10:55:01]: HTTP service is online.
07/16/09 10:55:01]: MYSQL service is online.
07/16/09 11:00:01]: LOAD 1.06 (status good)
07/16/09 11:00:01]: HTTP service is online.
07/16/09 11:00:01]: MYSQL service is online.
07/16/09 11:05:22]: LOAD 76.95 (status critical)
07/16/09 11:05:22]: load status criticalnon-essential services going down
As you can see, in 5 minutes server load grows from 1 to 77. When it happends, the problem usually repeats 2-3 times right after the stopped services are restarted. This happends momentally and I can not even check the top because server is almost frozen.

It happends only 1-2 times per month. I have only 1 website on this server and there are no scripts which are accessed only 1-2 times per month, so this couldn't be caused by bad scripts. I have one more server (Debian) with the same scripts and there are no problems like this. There are no other users on my server.

There are no spammers who are sending spam from my server.

I don't have any cron jobs that run only 1-2 times per month and may cause such a load.

I need any advice how to fix this problem and how to find out what causes it.

Thank you.
Old 07-16-2009, 05:25 AM   #2
LQ Veteran
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 15,647

Rep: Reputation: 2092Reputation: 2092Reputation: 2092Reputation: 2092Reputation: 2092Reputation: 2092Reputation: 2092Reputation: 2092Reputation: 2092Reputation: 2092Reputation: 2092
How often does this come up ... ?
loadavg is (runq+uninterruptible sleep). Generally that's disk I/O wait.
Try this when you have the "problem":
top -b -n 1 | awk '{if (NR <=7) print; else if ($8 == "D") {print; count++} } END {print "Total status D: "count}'
Old 07-16-2009, 05:28 AM   #3
Senior Member
Registered: Apr 2004
Location: Potchefstroom, South Africa
Distribution: Fedora 17 - 3.3.4-5.fc17.x86_64
Posts: 1,500

Rep: Reputation: 91
Not sure if this will help, but do you have AWStats or something similar running on that machine?

Alternatively, it should be simple to keep record of what IP addresses visit your site.

I. e. if you have a "load hit" taking place, wait for it to pass, then go and check AWStats or your IP log list.

I once had much the same problem with one of my sites - except its bandwidth use was running away completely. I used a log of visiting IP adresses and correlated that with the hourly bandwidth use. I quickly discovered the culprit IP - which turned out to be a University in another city that was excessively caching the site on their local intranet for their users.

So you should be able to do much the same, only you won't correlate against bandwidth use over time but against load over time. I'm betting you have some EXTERNAL server or script hitting your server to update a local cache on its side, or something similar. The fact that you state this happens only once or twice a month might point to something automatic, somewhere, really hitting your server hard.

I'm assuming the exact times vary? I. e. it is always at different times?

Last edited by rylan76; 07-20-2009 at 08:20 AM.
Old 07-16-2009, 07:13 AM   #4
LQ Newbie
Registered: Jul 2009
Posts: 6

Original Poster
Rep: Reputation: 0

Thank you for your replies.

No, I don't have AWStats installed or something similar.

I have this server since September 2008, but as I remember, the problems started in January. Unfortunately, I have deleted some warning messages from SIM, but here are 4 which I found in my mailbox.

Monday, January 12, 11:00
Monday, June 22, 14:00
Monday, July 13, 14:40
Thursday, July 16, 11:05

As you can see, 3 times of 4 the problem was on Monday. The times are different, but it always happends in time range from 10 am till 4 pm. Usually it happends not often than 2 times per month, but in July it happened already 2 times.

When the problem started, SIM reported about server loads like 20-25. Then it grew to ~30-35, but today the load was 77!

I know that some crawlers may cause high server loads, but this is a dating website with not so many information which is accessible without logging in. Crawlers can access only the startpage which shows the last discussions, last profiles, etc. At the same time lot of profiles are "open" and accessable by everyone without logging in. I have noticed that Google is indexing images from user profiles, but could it cause server loads like 70?


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
server load high graziano1968 Linux - General 5 03-12-2009 01:32 PM
Best Mail Server for high load mohakevin Linux - Server 8 01-05-2009 02:42 PM
Massive High Load on My server adam_blackice Linux - Security 11 12-12-2008 12:26 PM
Server high load and slow lavinya Linux - Server 20 12-01-2007 03:04 PM
Server Load Times High, help? Networks Linux - Newbie 6 05-05-2007 12:37 AM

All times are GMT -5. The time now is 04:16 PM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration