Hi,
I need help about the strange output of the top command
in my debian server. We see high load average with relatively
low cpu usage. Also iowait seems normal.
I think this is something about Java socket threading but
i don't know how to discover and fix exactly what is causing
the issue.

We are running a Java socket server on this Debian machine.
It's a Dual-Core AMD Opteron(tm) Processor 1210 processor
with 8GB RAM.

and java -version output:
java version "1.5.0_16"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b02)
Java HotSpot(TM) Server VM (build 1.5.0_16-b02, mixed mode)

and the top output:
top - 15:18:06 up 125 days, 2:41, 4 users, load average: 17.94, 15.81, 16.38
Tasks: 554 total, 2 running, 551 sleeping, 0 stopped, 1 zombie
Cpu(s): 8.9%us, 2.6%sy, 0.0%ni, 87.1%id, 0.0%wa, 0.7%hi, 0.7%si, 0.0%st
Mem: 8315176k total, 6809904k used, 1505272k free, 484384k buffers
Swap: 1879596k total, 0k used, 1879596k free, 3920524k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16654 root 16 0 1600m 287m 11m S 9 3.5 9:52.00 java
19461 www-data 15 0 36496 7464 3372 S 1 0.1 0:00.10 apache2
14302 root 16 0 2824 1448 864 S 1 0.0 0:57.34 top
4386 www-data 15 0 36752 6832 2744 S 1 0.1 0:00.19 apache2
16244 www-data 15 0 36752 6832 2744 S 1 0.1 0:00.14 apache2
31830 www-data 20 0 36752 6848 2760 S 1 0.1 0:00.17 apache2
21110 www-data 18 0 36752 6688 2736 S 1 0.1 0:00.05 apache2
21451 www-data 15 0 36516 6488 2736 S 1 0.1 0:00.03 apache2
23991 root 15 0 2696 1420 860 R 1 0.0 0:00.12 top

Any help will be appreciated.. thanks in advance

I don't think there's too much to worry about, there seems to be quite a few idle processes so you could possibly tune things a little. Maybe httpd is configured to spawn lots of children ... ?

cheer

High bandwith usage alone won't produce great processor power.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
update:it means that just dl/uploading without use of hard drive wont use much of cpu
example:router can take a lot of traffic on its slow processor

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When we shut down the java socket server, load average decreases to 2.0-3.0
so i thought it's about java. also we have another server (higher traffic)
with same apache configuration and it seems fine.

yooy, so it is about network latency? is there a way to measure it?

yooy, what the hell is that supposed to mean ?.
OP, try this and post the outputUse code tags when posting output ...

This is the output:

O.K., I'm confused - that should list all uninteruptible sleep tasks (which contribute to loadavg). I had a look at how loadavg is accumulated a while back - seemed straightforward. What kernel are you on ("uname -a") ?.

Kernel version: Linux 2.6.18-5-686-bigmem #1 SMP Sat Dec 1 23:58:00 UTC 2007 i686 GNU/Linux

Keep trying that - in a loop maybe, redirecting to a file. You might have a lot of short-lived processes. Dunno at this point.

I tried few times and once catch something like this:

That's one of the major problems with a tool like top - no history. You see what you see and that's it. PLUS you only see what top wants you to see.

You could always try out collect, either interactively or as a daemon. Dy default in daemon mode it samples everything but processes every 10 seconds and processes every 60 - extra overhead.

BUT if you really want to see what's happening over a relatively short period of time edit /etc/collectl.conf and add "-i1:1" to the line 'DaemonCommands' and that will monitor everything once a second.


"service collectl start" and let it run for a few minutes and then "service collectl stop". now play back the data it collected - too many options to list - but if you run:

collectl -p /var/log/collectl/filename -sxxx -oT

you'll see data for the subsystems specified with 'xxx' along with time stamps. 'c' will show CPU, 'd' disk, etc. "collectl --showsbsys" for a complete listing.

if you want to look at your top processes over time, which is what got me started, you can:

collectl -p filename --top

and you'll see the top 10 processes for every second!!! if you want to see more or less of them "collectl -x" and see the options for --top.

if you "collectl -p filename -sc --verbose -oT" you'll see the load averages along with the number of running processes AND the number of process creations/sec if that is a concern.

for more, just go to soureforge and look at the documentation http://collectl.sourceforge.net

have fun...

-mark

Who is this guy ???.
Seems to want to push collectl pretty hard.
.
.
.
.
Hey Mark, back again ...

I too like his little toy - unfortunately not everyone seems to want to use it.

hey back - yes I know you're a fan. I've seen previous posts by you recommending it. I do realize not everyone is on board with it but I also realize not everybody is convinced monitoring is important. I was talking to someone the other day who was a sar user. Nothing wrong with sar, just that people use a monitoring interval that's much too high. I suggested if that they at least drop the monitoring frequency down to 10 seconds as 10 minutes is pretty worthless. They said their vendor told them not to go below a minute and I told them their vendor is wrong! If collectl generates less that 0.1% cpu load running at 10 second monitoring and it's written in perl, SAR had got to have a lighter footprint. But some people just don't get it.

I would wonder why people don't use collectl:
- they don't believe in proactive monitoring
- they're happy with what the have
- they're scared of it

If the first, they're flat out wrong. If the second, that's fine as long as they monitor frequently. If the third I can help if they ask.

I believe EVERYONE should continuously monitor their systems at 5-15 second frequencies. There are a very few situations where I've seen monitoring have on impact on performance - applications that run at 100% cpuloads and are fine-grained parallel jobs running on 1000 cores or more. If you don't know what a find-grained parallel job is, you don't have to worry about collectl! First of all not many people run parallel jobs, let along fine-grained ones, and even less run on 1K cores or more. Even those who do run on that many cores still find a slight performance hit is worth it be able to have the data available if something goes wrong.

-mark