High loadavg, low CPU & iowait
 

I've got 4 servers, 2 of which are running Veritas Cluster Server (NFS only). The hardware on all 4 servers is identical. All 4 machines mount the NFS share exported by the active VCS node. The 2 machines running VCS (and VxVM) have LAs of 11+ (across 4 CPUs), whereas the other 2 have LAs of < 0.5. I can understand the active node having a high LA, but the inactive node should look more like the other 2 machines. The CPU utilization, swap utilization and iowait are almost 0.

Any ideas?

I've always wondered why I occassionally see higher load averages with low cpu utilization. As I understand it, load average is the number of processes waiting for a slot on the cpu (averaged over time). And if they're waiting for a slot it follows that this is because the cpu is busy. You'd think that if a process currently on the cpu was waiting on something else (io maybe), the kernel would turn the cpu over to a more deserving process ... but maybe not.

I'll be interested to see how others weigh in on this apparent paradox of high load average with high %idle and low %iowait (and low "everything else visible via 'top'"). The combination seems counterintuitive.