LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices



Reply
 
Search this Thread
Old 08-02-2009, 05:50 PM   #1
bagpussnz
Member
 
Registered: Aug 2003
Posts: 51

Rep: Reputation: 15
Responsive system/High load average and hanging ps


Hi,
I have a server running kernel...

2.6.8-24.14-smp #1 SMP Tue Mar 29 09:27:43 UTC 2005 x86_64 x86_64 x86_64 GNU/Linux

This server runs intense java build services (using maven).

What I am seeing is after a short while of booting, the machine starts getting a high load average (the highest I have seen is 3600).
However, the machine is still responsive.

When I do a ps -ef (or find through /proc) the command hangs (and cannot be terminated).

The only way to reboot is to pull the power.

Does this sound familiar to anyone? How can I diagnose?

top - 09:21:48 up 3 days, 1:20, 12 users, load average: 420.98, 414.12, 396.64
Tasks: 698 total, 1 running, 697 sleeping, 0 stopped, 0 zombie
Cpu(s): 21.9% us, 6.6% sy, 0.0% ni, 61.9% id, 9.3% wa, 0.0% hi, 0.3% si
Mem: 2055324k total, 2024640k used, 30684k free, 187044k buffers
Swap: 1052216k total, 280400k used, 771816k free, 1108756k cached


Regards,
Ian Collins.
 
Old 08-02-2009, 06:57 PM   #2
paulsm4
Guru
 
Registered: Mar 2004
Distribution: SusE 8.2
Posts: 5,863
Blog Entries: 1

Rep: Reputation: Disabled
Dude - you need more RAM. Fast!

If your system workload is such that the runlength queue chronically exceeds 2.0 ("two's a crowd" is a true statement!), that's an indication your system *might* need more horsepower.

Your runlength queue exceeds 400!

If "swap used" is chronically non-zero, that's a strong indication you need more RAM and/or need to throttle a "memory hog" process and/or need to break some of your "memory hogs" out to a separate server.

Moreover, "memory swapping" is certainly contributing to (and might in fact be the root cause) of your high run queue.

PS:
I have *never* seen a load average of "420.98". Never!

But I *have* seen systems visibly impacted with the load average as low as 2.0 - 5.0. Honest.

You need more RAM, you need to consider "throttling" your app(s) (perhaps with custom JVM switches), and should consider faster/bigger/more powerful systems, and you should also consider partitioning your workload across multiple servers.

IMHO .. PSM

Last edited by paulsm4; 08-02-2009 at 07:00 PM.
 
Old 06-01-2010, 03:09 PM   #3
Retrievil_Knievil
Member
 
Registered: Mar 2004
Location: Stavanger, Norway
Distribution: Gentoo, Slackware/SLAX, Knoppix, CentOS, IPCop & DSL
Posts: 138

Rep: Reputation: 21
Similar scenario - different solution

Hi,

Found this thread while I was having a similar problem, and found the solution to my own, anyway. I think such a high load average is more a sign of something being broken than just missing RAM.

On the system I was looking into the load average was well into the 40-50's, with only four cores, so something was up.

Turned out it was a nfs mount that was gone off line, and a lot of processes were stalled due to this halting any process that tries to list the drives or the folder containing the mount.

Upon rebooting the host with the unresponsive nfs mount and remounting it (the client complained that is was still mounted, but fixed the problem, and unmounting it would not work cleanly, since the mount was busy) everything went smoothly.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
load average and cpu usage too high, why could i do? v_fone Linux - Newbie 5 07-02-2009 04:17 AM
High load average for no apparent reason permalac Linux - Server 12 03-09-2009 12:13 PM
Load average stay as high as around 1.00 lawrence_lee_lee Linux - Software 2 09-10-2008 02:22 AM
high load average, low cpu usage ! jimmyjiang Red Hat 8 02-08-2008 01:28 AM
Why is my load average so high when comp. is idle? BrianK Linux - General 1 11-18-2005 01:25 AM


All times are GMT -5. The time now is 09:31 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration