LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 03-12-2009, 05:40 AM   #1
graziano1968
Member
 
Registered: Sep 2004
Posts: 223

Rep: Reputation: 30
server load high


Hello

this morning my server load was high , over 10 .

I was monitoring procs with
ps aux
and
top

I stopped apache and mysql. I noticed that the server load decreased to
3 , however it was not going down. Then I noticed that top was reporting and user usage of 60%-70% .

However I was not able to see any user process eating all cpu neither with top , neither with

ps aux | grep -v "root"

There could be some hidden process (not visible by top or ps) which can explain that 60% used by user ?
Looking at top procs list or ps there was nothing to explain that 60% us


Thank you!
 
Old 03-12-2009, 07:30 AM   #2
MensaWater
Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 6,009
Blog Entries: 5

Rep: Reputation: 787Reputation: 787Reputation: 787Reputation: 787Reputation: 787Reputation: 787Reputation: 787
Thumbs down

There are a few things that might cause this:

1) A poorly designed program or script that isn't allowing adequate time between runs of commands.
For example:
Code:
while true
do ls -lR /
done
The above script would still be doing the first ls -lR when it invoked the next one and then the next one might see the first 2 etc...
It is easy to solve this by simply adding a sleep statement sufficient to make it wait for completion of first command. Sometimes a simple "sleep 1" for a 1 second wait is sufficient. For something like the example I'd probably do a sleep 120 to give it 2 minutes to list all the files.

2) Similar to the above is the case where multiple users are running the same command that eats up all the same resources. For example if you had 100 users all running "top" at the same time you'd see an issue because they'd all be competing for the same information.

3) Less common: I've seen on a couple of Dell servers with PERC cards that the PERC cards are very sensitive to heat. When they heat up they might cause a file/directory to lock. From that point on all attempts to access that file/directory hang because they are trying to complete and this will increase the cpu load ad infinitum as new processes are kicked off (e.g. locate command from cron). The only solution for this is to reboot the server to clear the lock. Of course then you also have to address the heat issue to prevent recurrence.
Note that Dell doesn't admit there is any heat sensitivity to the PERC cards so my comments are based on observation. I've seen this even when the system's diagnostics don't show any heat threshold being exceeded but the PERC card has no heat sensor. The PERC is made by LSI so presumably this could be an issue in any other LSI controller based system like the Dell PowerEdge machines.
 
Old 03-12-2009, 10:02 AM   #3
graziano1968
Member
 
Registered: Sep 2004
Posts: 223

Original Poster
Rep: Reputation: 30
In case 1) why I can't see this script running ps aux ?

In case 2) , no I can't see all this procs investigating with ps

3) is not my case.
 
Old 03-12-2009, 11:40 AM   #4
jamescondron
Member
 
Registered: Jul 2007
Location: Scunthorpe, UK
Distribution: Ubuntu 8.10; Gentoo; Debian Lenny
Posts: 961

Rep: Reputation: 69
Whats your wa level on top showing? My load averages were doing similar a while ago, turns out the hard drive was failing. Of course, did you take a look at incoming connections, or open files? There could be all kinds of things happening inside or outside of your box. I find iftop is a good tool for monitoring connections and incoming data. lsof, of course, is to list open files. Track back from either of those and look for a rogue process.
 
Old 03-12-2009, 11:46 AM   #5
graziano1968
Member
 
Registered: Sep 2004
Posts: 223

Original Poster
Rep: Reputation: 30
Quote:
Originally Posted by jamescondron View Post
Whats your wa level on top showing? My load averages were doing similar a while ago, turns out the hard drive was failing. Of course, did you take a look at incoming connections, or open files? There could be all kinds of things happening inside or outside of your box. I find iftop is a good tool for monitoring connections and incoming data. lsof, of course, is to list open files. Track back from either of those and look for a rogue process.
it's not the case too , wa was under 5% I'm sure of this.
I experienced too an hard drive failure some month ago where wa was always at the top, it's not this case (luckly!).

Thanks for iftop I will look it !!

Last edited by graziano1968; 03-12-2009 at 11:52 AM.
 
Old 03-12-2009, 01:32 PM   #6
MensaWater
Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 6,009
Blog Entries: 5

Rep: Reputation: 787Reputation: 787Reputation: 787Reputation: 787Reputation: 787Reputation: 787Reputation: 787
Quote:
Originally Posted by graziano1968 View Post
In case 1) why I can't see this script running ps aux ?

In case 2) , no I can't see all this procs investigating with ps

3) is not my case.
If it were a script you'd not see the script running but rather the commands within the script at the time they were running. That is to say for my example you'd see the "ls -l" rather than whatever you'd named the script. It is a little difficult to track this kind of thing down because usually you'll have multiple commands running in a script so at any given moment the commands in the process tree may be different.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Best Mail Server for high load mohakevin Linux - Server 8 01-05-2009 02:42 PM
Massive High Load on My server adam_blackice Linux - Security 11 12-12-2008 12:26 PM
Server high load and slow lavinya Linux - Server 20 12-01-2007 03:04 PM
Server Load Times High, help? Networks Linux - Newbie 6 05-05-2007 12:37 AM
High server load problem eagletalontim Linux - General 5 01-12-2006 10:42 AM


All times are GMT -5. The time now is 12:27 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration