LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 07-16-2009, 06:03 AM   #1
lpa
LQ Newbie
 
Registered: Jul 2009
Posts: 6

Rep: Reputation: 0
Extremaly high server load 2-3 times per month


Hello,

I have the following problem. Usually the average load of my server (CentOS) is about 0.5 - 2. Every month there are 1-2 black days when happends the following:

PHP Code:
SIM Log:
[
07/16/09 10:40:01]: MYSQL service is online.
[
07/16/09 10:45:01]: LOAD 0.58 (status good)
[
07/16/09 10:45:01]: HTTP service is online.
[
07/16/09 10:45:01]: MYSQL service is online.
[
07/16/09 10:50:01]: LOAD 1.50 (status good)
[
07/16/09 10:50:01]: HTTP service is online.
[
07/16/09 10:50:01]: MYSQL service is online.
[
07/16/09 10:55:01]: LOAD 1.12 (status good)
[
07/16/09 10:55:01]: HTTP service is online.
[
07/16/09 10:55:01]: MYSQL service is online.
[
07/16/09 11:00:01]: LOAD 1.06 (status good)
[
07/16/09 11:00:01]: HTTP service is online.
[
07/16/09 11:00:01]: MYSQL service is online.
[
07/16/09 11:05:22]: LOAD 76.95 (status critical)
[
07/16/09 11:05:22]: load status criticalnon-essential services going down
As you can see, in 5 minutes server load grows from 1 to 77. When it happends, the problem usually repeats 2-3 times right after the stopped services are restarted. This happends momentally and I can not even check the top because server is almost frozen.

It happends only 1-2 times per month. I have only 1 website on this server and there are no scripts which are accessed only 1-2 times per month, so this couldn't be caused by bad scripts. I have one more server (Debian) with the same scripts and there are no problems like this. There are no other users on my server.

There are no spammers who are sending spam from my server.

I don't have any cron jobs that run only 1-2 times per month and may cause such a load.


I need any advice how to fix this problem and how to find out what causes it.

Thank you.
 
Old 07-16-2009, 06:25 AM   #2
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 14,832

Rep: Reputation: 1820Reputation: 1820Reputation: 1820Reputation: 1820Reputation: 1820Reputation: 1820Reputation: 1820Reputation: 1820Reputation: 1820Reputation: 1820Reputation: 1820
How often does this come up ... ?
loadavg is (runq+uninterruptible sleep). Generally that's disk I/O wait.
Try this when you have the "problem":
Code:
top -b -n 1 | awk '{if (NR <=7) print; else if ($8 == "D") {print; count++} } END {print "Total status D: "count}'
 
Old 07-16-2009, 06:28 AM   #3
rylan76
Senior Member
 
Registered: Apr 2004
Location: Potchefstroom, South Africa
Distribution: Fedora 17 - 3.3.4-5.fc17.x86_64
Posts: 1,487

Rep: Reputation: 88
Not sure if this will help, but do you have AWStats or something similar running on that machine?

Alternatively, it should be simple to keep record of what IP addresses visit your site.

I. e. if you have a "load hit" taking place, wait for it to pass, then go and check AWStats or your IP log list.

I once had much the same problem with one of my sites - except its bandwidth use was running away completely. I used a log of visiting IP adresses and correlated that with the hourly bandwidth use. I quickly discovered the culprit IP - which turned out to be a University in another city that was excessively caching the site on their local intranet for their users.

So you should be able to do much the same, only you won't correlate against bandwidth use over time but against load over time. I'm betting you have some EXTERNAL server or script hitting your server to update a local cache on its side, or something similar. The fact that you state this happens only once or twice a month might point to something automatic, somewhere, really hitting your server hard.

I'm assuming the exact times vary? I. e. it is always at different times?

Last edited by rylan76; 07-20-2009 at 09:20 AM.
 
Old 07-16-2009, 08:13 AM   #4
lpa
LQ Newbie
 
Registered: Jul 2009
Posts: 6

Original Poster
Rep: Reputation: 0
Hello,

Thank you for your replies.

No, I don't have AWStats installed or something similar.

I have this server since September 2008, but as I remember, the problems started in January. Unfortunately, I have deleted some warning messages from SIM, but here are 4 which I found in my mailbox.

Monday, January 12, 11:00
Monday, June 22, 14:00
Monday, July 13, 14:40
Thursday, July 16, 11:05

As you can see, 3 times of 4 the problem was on Monday. The times are different, but it always happends in time range from 10 am till 4 pm. Usually it happends not often than 2 times per month, but in July it happened already 2 times.

When the problem started, SIM reported about server loads like 20-25. Then it grew to ~30-35, but today the load was 77!

I know that some crawlers may cause high server loads, but this is a dating website with not so many information which is accessible without logging in. Crawlers can access only the startpage which shows the last discussions, last profiles, etc. At the same time lot of profiles are "open" and accessable by everyone without logging in. I have noticed that Google is indexing images from user profiles, but could it cause server loads like 70?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
server load high graziano1968 Linux - General 5 03-12-2009 02:32 PM
Best Mail Server for high load mohakevin Linux - Server 8 01-05-2009 03:42 PM
Massive High Load on My server adam_blackice Linux - Security 11 12-12-2008 01:26 PM
Server high load and slow lavinya Linux - Server 20 12-01-2007 04:04 PM
Server Load Times High, help? Networks Linux - Newbie 6 05-05-2007 01:37 AM


All times are GMT -5. The time now is 06:37 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration