LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 03-11-2010, 09:12 AM   #1
lancherider
LQ Newbie
 
Registered: Apr 2006
Location: Boulder, CO
Distribution: RHEL, CentOS, Ubuntu
Posts: 21

Rep: Reputation: 0
High load average, low cpu usage on CentOS 5.4 64-bit


I know this has been asked before, but I haven't found anything that matches my scenario. I have a CentOS 5.4 server that is experiencing escalating load average without any other symptoms. The cpu's are nearly idle and the system is responsive, yet the load average over the past 48 hours has climbed to ~41. OK, relevant details. This system has 2 2GHZ quad core Intel Xeons, 32GB RAM. Here is load average:

Code:
[root@system ~]# uptime
 08:03:49 up 5 days, 21:26,  1 user,  load average: 41.00, 41.01, 40.94
Here is output of top, sorted by cpu time:

Code:
top - 08:06:39 up 5 days, 21:29,  1 user,  load average: 41.00, 41.00, 40.94
Tasks: 392 total,   1 running, 391 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  32962516k total, 23681516k used,  9281000k free,   822964k buffers
Swap:  4192924k total,        0k used,  4192924k free,  9317960k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+    TIME COMMAND                                                      
11687 tomcat    18   0 12.6g  11g 9.8m S  0.0 36.4  60:02.63  60:02 jsvc                                                          
11548 tomcat    18   0 1018m 132m 8780 S  0.0  0.4  30:23.18  30:23 jsvc                                                                                                                  
11806 root      18   0 10228  684  588 S  0.0  0.0   1:09.15   1:09 hald-addon-stor                                               
11267 root      15   0     0    0    0 S  0.0  0.0   1:05.93   1:05 cvfs_sockfsm                                                  
11220 root      15   0     0    0    0 S  0.0  0.0   1:04.30   1:04 cvfs_sockfsm                                                  
11065 root      15   0     0    0    0 S  0.0  0.0   1:03.82   1:03 cvfs_sockfsm                                                  
11314 root      15   0     0    0    0 S  0.0  0.0   1:03.71   1:03 cvfs_sockfsm                                                  
11176 root      15   0     0    0    0 S  0.0  0.0   0:58.83   0:58 cvfs_sockfsm                                                  
11130 root      15   0     0    0    0 S  0.0  0.0   0:57.04   0:57 cvfs_sockfsm                                                  
11780 haldaemo  15   0 37988  10m 1580 S  0.0  0.0   0:56.64   0:56 hald                                                          
11048 root      15   0     0    0    0 S  0.0  0.0   0:22.51   0:22 cvfs_ioc                                                      
11597 tomcat    18   0 1117m 212m 8752 S  0.0  0.7   0:20.30   0:20 jsvc                                                          
11642 tomcat    18   0  848m 132m 8664 S  0.0  0.4   0:14.55   0:14 jsvc                                                          
11414 srm       18   0  885m 160m 8780 S  0.0  0.5   0:12.30   0:12 java                                                          
11297 root      15   0     0    0    0 S  0.0  0.0   0:10.87   0:10 mi_unsolicitq                                                 
11293 root      15   0     0    0    0 S  0.0  0.0   0:10.82   0:10 mi_unsolicitq                                                 
11089 root      15   0     0    0    0 S  0.0  0.0   0:10.74   0:10 mi_unsolicitq                                                 
11092 root      15   0     0    0    0 S  0.0  0.0   0:10.73   0:10 mi_unsolicitq                                                 
11275 root      15   0     0    0    0 S  0.0  0.0   0:10.71   0:10 cvfsbufiod                                                    
11090 root      15   0     0    0    0 S  0.0  0.0   0:10.69   0:10 mi_unsolicitq                                                 
11336 root      15   0     0    0    0 S  0.0  0.0   0:10.66   0:10 mi_unsolicitq                                                 
11270 root      15   0     0    0    0 S  0.0  0.0   0:10.63   0:10 cvfsbufiod                                                    
11247 root      15   0     0    0    0 S  0.0  0.0   0:10.62   0:10 mi_unsolicitq                                                 
11241 root      15   0     0    0    0 S  0.0  0.0   0:10.61   0:10 mi_unsolicitq                                                 
11271 root      15   0     0    0    0 S  0.0  0.0   0:10.60   0:10 cvfsbufiod                                                    
11272 root      15   0     0    0    0 S  0.0  0.0   0:10.60   0:10 cvfsbufiod                                                    
11289 root      15   0     0    0    0 S  0.0  0.0   0:10.60   0:10 mi_unsolicitq                                                 
11273 root      15   0     0    0    0 S  0.0  0.0   0:10.56   0:10 cvfsbufiod                                                    
11276 root      15   0     0    0    0 S  0.0  0.0   0:10.54   0:10 cvfsbufiod                                                    
11294 root      15   0     0    0    0 S  0.0  0.0   0:10.52   0:10 mi_unsolicitq                                                 
11269 root      15   0     0    0    0 S  0.0  0.0   0:10.49   0:10 cvfsbufiod
I didn't post vmstat and iostat, because it's quite a long list (system is attached to a SAN and each disk shows up in the output), but there doesn't seem to be anything to be gleaned from that output either. So my question is, 1) do I need to worry and 2) what is causing the high load average? I've been waiting to see if it will clear up on it's own, since it's not affecting overall system performance, but it just keeps climbing.
 
Old 03-11-2010, 10:30 AM   #2
smoker
Senior Member
 
Registered: Oct 2004
Distribution: Fedora Core 4, 12, 13, 14, 15, 17
Posts: 2,279

Rep: Reputation: 248Reputation: 248Reputation: 248
Hi,

Here's a good thread :
http://osdir.com/ml/centos/2009-12/msg01346.html

the resolution is here :
http://osdir.com/ml/centos/2009-12/msg01410.html

Last edited by smoker; 03-11-2010 at 10:32 AM.
 
Old 03-11-2010, 03:25 PM   #3
lancherider
LQ Newbie
 
Registered: Apr 2006
Location: Boulder, CO
Distribution: RHEL, CentOS, Ubuntu
Posts: 21

Original Poster
Rep: Reputation: 0
Thanks, but exim is not even installed. Postfix is running, but the queue is totally empty, so it doesn't seem like that is the culprit. I'm going to proceed on the assumption that this is strange behavior, but that's all it is. Doesn't seem to be affecting system performance overall, so the main threat seems to be filling up the process table, but I've got a ways to go before I really need to worry about that.
 
Old 03-11-2010, 05:33 PM   #4
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,276

Rep: Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028Reputation: 1028
Loadave is a different metric under Linux than classic Unix - it includes tasks in uninteruptable sleep.
If you can say "Doesn't seem to be affecting system performance overall" I guess that means your service to your users is acceptable. If so, go find something else to worry about.
Run this to find the errant tasks - could be I/O waits (doesn't look like it), or buggy software that is forgetting about children. Or maybe you have a looping script, or cron, or ...
Code:
top -b -n 1 | awk '{if (NR <=7) print; else if ($8 == "D") {print; count++} } END {print "Total status D: "count}'
 
Old 06-01-2010, 04:08 PM   #5
Retrievil_Knievil
Member
 
Registered: Mar 2004
Location: Stavanger, Norway
Distribution: Gentoo, Slackware/SLAX, Knoppix, CentOS, IPCop & DSL
Posts: 138

Rep: Reputation: 21
Mounts?

Recently had this issue due to missing mounts. Do you have any nfs mounts that are unresponsive, perhaps?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
load average and cpu usage too high, why could i do? v_fone Linux - Newbie 5 07-02-2009 03:17 AM
high load average, low cpu usage ! jimmyjiang Red Hat 8 02-08-2008 12:28 AM
Qmail problems - CPU load average rising too high xbaez Linux - Software 0 11-16-2005 12:23 PM
RH8 Load Average High - No CPU Utilization jj91709 Red Hat 2 08-29-2004 12:28 AM
Load average stuck at 7.00, CPU usage ~ 0.1%, what gives? BrianK Linux - General 4 02-16-2004 08:45 PM


All times are GMT -5. The time now is 09:05 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration