Load is very high but CPU usage is almost zero in top!
 

I am facing a strange problem. The load on my server is getting very high as you can see in the top output but the CPU usage is almost zero.

I have noticed that the load is getting high whenever I start apache2.

The server has apache2, tomcat, mysql

Please help

top - 08:29:51 up 63 days, 22:53, 1 user, load average: 12.95, 10.61, 8.35
Tasks: 163 total, 1 running, 162 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.7%us, 0.6%sy, 0.0%ni, 98.4%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 33023640k total, 19154120k used, 13869520k free, 373012k buffers
Swap: 999992k total, 676k used, 999316k free, 13818440k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 10312 748 620 S 0 0.0 1:54.74 init
2 root 15 -5 0 0 0 S 0 0.0 3:27.03 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:01.66 migration/0
4 root 15 -5 0 0 0 S 0 0.0 0:27.74 ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 3365:16 watchdog/0
6 root RT -5 0 0 0 S 0 0.0 0:08.92 migration/1

And ?.
Does it affect your ability to service your users ?. Doesn't look like any I/O wait there, but you could check with iostat or similar.

Yes it affect the users and load is always increasing

the iostat is:

avg-cpu: %user %nice %system %iowait %steal %idle
0,73 0,00 0,59 0,33 0,00 98,36

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 2,88 47,34 136,57 263814359 761116922
sda1 0,00 0,01 0,00 79062 146
sda2 0,00 0,00 0,00 784 1352
sda3 2,88 47,32 136,57 263734297 761115424


When load was 32.39

top - 21:40:09 up 64 days, 12:03, 1 user, load average: 32.39, 24.65, 14.31
Tasks: 188 total, 1 running, 187 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.1%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 33023640k total, 20265148k used, 12758492k free, 389448k buffers
Swap: 999992k total, 676k used, 999316k free, 14192196k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
27460 root 20 0 18956 1336 940 R 1 0.0 0:00.08 top
1 root 20 0 10312 748 620 S 0 0.0 1:55.09 init
2 root 15 -5 0 0 0 S 0 0.0 3:27.03 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:01.66 migration/0
4 root 15 -5 0 0 0 S 0 0.0 0:27.74 ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 3381:13 watchdog/0
6 root RT -5 0 0 0 S 0 0.0 0:08.94 migration/1
7 root 15 -5 0 0 0 S 0 0.0 0:05.14 ksoftirqd/1

In Linux loadavg is the count of runnable tasks plus those in uninterruptible sleep. The latter are usually waiting on disk I/O, but can be generated by a programming bug - maybe a bad script, who knows. You don't show any indication of I/O wait. run this to see the processes in uninterruptible wait