High load average, low cpu usage on CentOS 5.4 64-bit

lancherider · 03-11-2010, 09:12 AM

I know this has been asked before, but I haven't found anything that matches my scenario. I have a CentOS 5.4 server that is experiencing escalating load average without any other symptoms. The cpu's are nearly idle and the system is responsive, yet the load average over the past 48 hours has climbed to ~41. OK, relevant details. This system has 2 2GHZ quad core Intel Xeons, 32GB RAM. Here is load average:

Code:

[root@system ~]# uptime
 08:03:49 up 5 days, 21:26,  1 user,  load average: 41.00, 41.01, 40.94

Here is output of top, sorted by cpu time:

Code:

top - 08:06:39 up 5 days, 21:29,  1 user,  load average: 41.00, 41.00, 40.94
Tasks: 392 total,   1 running, 391 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  32962516k total, 23681516k used,  9281000k free,   822964k buffers
Swap:  4192924k total,        0k used,  4192924k free,  9317960k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+    TIME COMMAND                                                      
11687 tomcat    18   0 12.6g  11g 9.8m S  0.0 36.4  60:02.63  60:02 jsvc                                                          
11548 tomcat    18   0 1018m 132m 8780 S  0.0  0.4  30:23.18  30:23 jsvc                                                                                                                  
11806 root      18   0 10228  684  588 S  0.0  0.0   1:09.15   1:09 hald-addon-stor                                               
11267 root      15   0     0    0    0 S  0.0  0.0   1:05.93   1:05 cvfs_sockfsm                                                  
11220 root      15   0     0    0    0 S  0.0  0.0   1:04.30   1:04 cvfs_sockfsm                                                  
11065 root      15   0     0    0    0 S  0.0  0.0   1:03.82   1:03 cvfs_sockfsm                                                  
11314 root      15   0     0    0    0 S  0.0  0.0   1:03.71   1:03 cvfs_sockfsm                                                  
11176 root      15   0     0    0    0 S  0.0  0.0   0:58.83   0:58 cvfs_sockfsm                                                  
11130 root      15   0     0    0    0 S  0.0  0.0   0:57.04   0:57 cvfs_sockfsm                                                  
11780 haldaemo  15   0 37988  10m 1580 S  0.0  0.0   0:56.64   0:56 hald                                                          
11048 root      15   0     0    0    0 S  0.0  0.0   0:22.51   0:22 cvfs_ioc                                                      
11597 tomcat    18   0 1117m 212m 8752 S  0.0  0.7   0:20.30   0:20 jsvc                                                          
11642 tomcat    18   0  848m 132m 8664 S  0.0  0.4   0:14.55   0:14 jsvc                                                          
11414 srm       18   0  885m 160m 8780 S  0.0  0.5   0:12.30   0:12 java                                                          
11297 root      15   0     0    0    0 S  0.0  0.0   0:10.87   0:10 mi_unsolicitq                                                 
11293 root      15   0     0    0    0 S  0.0  0.0   0:10.82   0:10 mi_unsolicitq                                                 
11089 root      15   0     0    0    0 S  0.0  0.0   0:10.74   0:10 mi_unsolicitq                                                 
11092 root      15   0     0    0    0 S  0.0  0.0   0:10.73   0:10 mi_unsolicitq                                                 
11275 root      15   0     0    0    0 S  0.0  0.0   0:10.71   0:10 cvfsbufiod                                                    
11090 root      15   0     0    0    0 S  0.0  0.0   0:10.69   0:10 mi_unsolicitq                                                 
11336 root      15   0     0    0    0 S  0.0  0.0   0:10.66   0:10 mi_unsolicitq                                                 
11270 root      15   0     0    0    0 S  0.0  0.0   0:10.63   0:10 cvfsbufiod                                                    
11247 root      15   0     0    0    0 S  0.0  0.0   0:10.62   0:10 mi_unsolicitq                                                 
11241 root      15   0     0    0    0 S  0.0  0.0   0:10.61   0:10 mi_unsolicitq                                                 
11271 root      15   0     0    0    0 S  0.0  0.0   0:10.60   0:10 cvfsbufiod                                                    
11272 root      15   0     0    0    0 S  0.0  0.0   0:10.60   0:10 cvfsbufiod                                                    
11289 root      15   0     0    0    0 S  0.0  0.0   0:10.60   0:10 mi_unsolicitq                                                 
11273 root      15   0     0    0    0 S  0.0  0.0   0:10.56   0:10 cvfsbufiod                                                    
11276 root      15   0     0    0    0 S  0.0  0.0   0:10.54   0:10 cvfsbufiod                                                    
11294 root      15   0     0    0    0 S  0.0  0.0   0:10.52   0:10 mi_unsolicitq                                                 
11269 root      15   0     0    0    0 S  0.0  0.0   0:10.49   0:10 cvfsbufiod

I didn't post vmstat and iostat, because it's quite a long list (system is attached to a SAN and each disk shows up in the output), but there doesn't seem to be anything to be gleaned from that output either. So my question is, 1) do I need to worry and 2) what is causing the high load average? I've been waiting to see if it will clear up on it's own, since it's not affecting overall system performance, but it just keeps climbing.

smoker · 03-11-2010, 10:30 AM

Hi,

Here's a good thread :
http://osdir.com/ml/centos/2009-12/msg01346.html

the resolution is here :
http://osdir.com/ml/centos/2009-12/msg01410.html

lancherider · 03-11-2010, 03:25 PM

Thanks, but exim is not even installed. Postfix is running, but the queue is totally empty, so it doesn't seem like that is the culprit. I'm going to proceed on the assumption that this is strange behavior, but that's all it is. Doesn't seem to be affecting system performance overall, so the main threat seems to be filling up the process table, but I've got a ways to go before I really need to worry about that.

syg00 · 03-11-2010, 05:33 PM

Loadave is a different metric under Linux than classic Unix - it includes tasks in uninteruptable sleep.
If you can say "Doesn't seem to be affecting system performance overall" I guess that means your service to your users is acceptable. If so, go find something else to worry about.
Run this to find the errant tasks - could be I/O waits (doesn't look like it), or buggy software that is forgetting about children. Or maybe you have a looping script, or cron, or ...

Code:

top -b -n 1 | awk '{if (NR <=7) print; else if ($8 == "D") {print; count++} } END {print "Total status D: "count}'

Retrievil_Knievil · 06-01-2010, 04:08 PM

Recently had this issue due to missing mounts. Do you have any nfs mounts that are unresponsive, perhaps?