Hi,

My server (PC with CentOS 5) is out of resources, consuming 100% of the cpu, but the list of processes do not add up to anything close to that, they add up to around 15%. What is using the other 85% of the cpu?

I'll appreciate any hints or ideas.

Here is a dump of the top command:
top - 14:57:20 up 5 days, 21:06, 2 users, load average: 4.04, 3.65, 2.53
Tasks: 69 total, 5 running, 64 sleeping, 0 stopped, 0 zombie
Cpu(s): 56.0%us, 44.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 499352k total, 458176k used, 41176k free, 124836k buffers
Swap: 2048276k total, 25016k used, 2023260k free, 193796k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9238 apache 25 0 26604 12m 3136 S 4.7 2.5 0:20.66 httpd
10855 apache 25 0 25724 11m 3132 R 4.3 2.4 0:20.38 httpd
9244 apache 25 0 26632 12m 3124 R 3.7 2.6 0:22.69 httpd
9243 apache 23 0 26124 12m 3144 S 1.0 2.5 0:30.22 httpd
1 root 15 0 2036 124 96 S 0.0 0.0 0:00.58 init
2 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/0
4 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
5 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/0
6 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 khelper
7 root 16 -5 0 0 0 S 0.0 0.0 0:00.00 kthread
10 root 10 -5 0 0 0 S 0.0 0.0 0:05.37 kblockd/0
11 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 kacpid
98 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/0
101 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 khubd
103 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kseriod
168 root 17 -5 0 0 0 S 0.0 0.0 1:12.41 kswapd0
169 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 aio/0
329 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 kpsmoused
346 root 10 -5 0 0 0 S 0.0 0.0 5:46.86 kjournald
373 root 10 -5 0 0 0 S 0.0 0.0 0:00.03 kauditd
407 root 20 -4 2124 56 52 S 0.0 0.0 0:00.26 udevd
714 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kedac
1078 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 kgameportd
1273 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 kmirrord
1293 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kjournald
1295 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kjournald
1815 root 16 -3 12048 288 204 S 0.0 0.1 0:00.93 auditd
1817 root 12 -3 9616 888 428 S 0.0 0.2 0:01.32 python
1876 rpc 25 0 1772 60 56 S 0.0 0.0 0:00.00 portmap
1892 dbus 23 0 2712 336 252 S 0.0 0.1 0:00.03 dbus-daemon
1907 root 18 0 1640 64 60 S 0.0 0.0 0:00.00 acpid
1925 root 16 0 5172 300 200 S 0.0 0.1 0:01.15 sshd
2065 root 18 0 5228 204 136 S 0.0 0.0 0:00.10 crond
2094 root 18 0 2200 176 136 S 0.0 0.0 0:00.03 atd
2117 root 15 0 32120 15m 2140 S 0.0 3.2 3:12.26 yum-updatesd
2132 haldaemo 18 0 5312 556 236 S 0.0 0.1 0:00.84 hald
2133 root 25 0 3104 164 160 S 0.0 0.0 0:00.00 hald-runner
2139 haldaemo 25 0 1968 88 84 S 0.0 0.0 0:00.00 hald-addon-acpi
2144 haldaemo 17 0 1968 92 88 S 0.0 0.0 0:00.00 hald-addon-keyb
2183 root 18 0 1952 280 128 S 0.0 0.1 0:00.12 smartd
2188 root 15 0 10420 1536 652 S 0.0 0.3 0:00.39 miniserv.pl
2193 root 15 0 1624 60 56 S 0.0 0.0 0:00.00 mingetty
2194 root 15 0 1624 60 56 S 0.0 0.0 0:00.00 mingetty
2195 root 15 0 1628 60 56 S 0.0 0.0 0:00.00 mingetty

hey
as I can see you have a load on the memory too, 56% of the cpu usage is taken for user space and 44 is taken for the system space.
try to stop some services like httpd and see the result.

Regards
Mhm

But the question is that httpd is only taking up 13.7% of the cpu. Shouldn't it show there if it were taking up more cpu usage?

How the obvious answer - something that's not a "process" ???.
Start turning off what you don't need - SMP, SMC, ACPI ... whatever.
Turn one off, check if it helps, then goto the next. Basic problem resolution.

Personally I'd probably start with ACPI - do a search on "max_cstate".

And how do I turn off things that are not processes?!
And how do I know what I do need (I don't really know what SMP, SMC, or ACPI are!...)

Thanks for your help.

Take note of the following "cat /sys/module/processor/parameters/max_cstate". Then I'd be inclined to maybe try "echo 1 > /sys/module/processor/parameters/max_cstate" and see if it has any effect.
You can always set it back to the original value or reboot to restore the value.

It's entirely possible that the CPU is being eaten up by swapping or waiting on other I/O. It looks like some amount of swap space is being used. Try running "vmstat 1 30" and see what the I/O load is. You can also get the relative loads from sar if you have that installed. If httpd is serving up a bunch of pages it's likely hitting the disk quite hard and that's a more likely explanation than something like ACPI IMO.

Not likely - that amount of swap usage over 5 days certainly isn't massive.
No mention of kswapd - which *is* a process you can see.
I/O wait contributes to the load averages, not the sys and usr CPU% - you'd see it in the other percentages.

Can't hurt to check it out though.

cat /sys/module/processor/parameters/max_cstate gives 8
Is that too big? What is that exactly?

The question is that the web server is not serving pages yet. Nobody is hitting it.

But regardless of the specific cause of the high load on that server, can anyone explain why the sum of cpu usages for all the processes reported by 'top' does not add up to the total cpu usage reported by the same utility?

Thanks a lot.

I'm guessing you have a hit a bug in kernel level code.
As I suggested, hit google on max_cstate. Others have seen similar.