Load average question
 

I thought I had a good understanding of load average. The number represents processes waiting for service by the cpu. It equates to customers in line at a movie theater. If there are 2 or less in line then your doing alright. Anything over 2 for a sustained period is not good. The higher you go the worse the performace will get all around.

Today I see a server with a sustained load average of 4.7. This is on a 4 processor box. Here are the numbers from top:

10:05:32 up 31 days, 15:33, 1 user, load average: 4.05, 4.10, 4.10
172 processes: 171 sleeping, 1 running, 0 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 1.3% 0.0% 0.6% 0.0% 0.0% 0.1% 97.7%
cpu00 1.1% 0.0% 0.5% 0.0% 0.0% 0.1% 98.0%
cpu01 2.1% 0.0% 0.3% 0.1% 0.1% 0.0% 97.0%
cpu02 1.1% 0.0% 1.5% 0.0% 0.0% 0.1% 97.0%
cpu03 0.7% 0.0% 0.1% 0.0% 0.0% 0.1% 98.8%
Mem: 7736296k av, 7185856k used, 550440k free, 0k shrd, 282884k buff
1703364k actv, 3965600k in_d, 169376k in_c
Swap: 4192924k av, 356k used, 4192568k free 5845148k cached

in case the numbers don't line up correctly ....all the system processors are 97 or 98% idle. Why in the world would I have a 4+ load average with idle cpu's? :confused:

You've only considered part of the equation - the loadaverage includes task_running (on a CPU or ready), but also task_uninterruptable. This should (generally) be something like disk wait, but poorly designed code could put itself in this state for extended periods.
Shouldn't, but could.