High load issue, need help

newlinuxnewbie · 09-16-2006, 11:06 AM

P4 2.53GHz, 1G RAM, 80G SATA HD, Cent OS 4.4

Load is at about 150, 300M in swap, 6% CPU usage. I run iostat and iowait is at about 7%. There are a also lot of apache processes in the D state, how do I fix this?

Can someone tell me what the wa in here stands for?
Cpu(s): 3.3% us, 3.6% sy, 0.0% ni, 0.0% id, 92.7% wa, 0.0% hi, 0.0% si

Another question, how do I output top and ps to a file?

Thanks!

bulliver · 09-16-2006, 02:04 PM

Quote:

There are a also lot of apache processes in the D state, how do I fix this?

D is DNS lookup. Is your DNS set up properly? Are you logging with reverse DNS lookup? If so you should almost certainly turn it off.

Quote:

Can someone tell me what the wa in here stands for?

AFAIK it means the CPU is 'waiting' for another process

Quote:

Another question, how do I output top and ps to a file?

$ ps aux > ps.txt
$ top -n1 > top.txt

generic_user · 09-16-2006, 02:51 PM

Quote:

Originally Posted by bulliver

D is DNS lookup. Is your DNS set up properly? Are you logging with reverse DNS lookup? If so you should almost certainly turn it off.

Say what?

From 'man ps'

Quote:

D Uninterruptible sleep (usually IO)

Quote:

AFAIK it means the CPU is 'waiting' for another process

It's the system waiting for IO's to complete, normally due to the filesystem not being able to keep up.

iostat -xd 1

will show you, among other things, the utilization of your hd. I'm willing to bet you'll see 100% for your drive.

bulliver · 09-16-2006, 02:55 PM

My bad, I was thinking of "Apache Server Status":

Quote:

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

Quote:

It's the system waiting for IO's to complete, normally due to the filesystem not being able to keep up

Well, I said AFAIK, and since they don't document it in the manpage...

syg00 · 09-16-2006, 05:24 PM

Note that the "D" indicates the state not the reason.
It's usually I/O wait (as the cited manpage says), but the code can put itself into uninteruptable for other reasons.
If, as the OP said, iostat showed nothing, maybe you have a bug, but maybe you should have a look for looping scripts, those sort of things.

newlinuxnewbie · 09-17-2006, 07:20 AM

Still having issues, a lot of apache process are in D state. How do I reduce this?

Quote:

Originally Posted by bulliver

$ ps aux > ps.txt
$ top -n1 > top.txt

Says command not found? Is that what I should type in ssh?

newlinuxnewbie · 09-17-2006, 08:57 AM

I found out the server is pushing about 80mbit average, could this be the reason for iowait to be so high? Load average is about 200 now

generic_user · 09-17-2006, 07:17 PM

Quote:

Originally Posted by newlinuxnewbie

I found out the server is pushing about 80mbit average, could this be the reason for iowait to be so high? Load average is about 200 now

You still haven't posted the output of iostat -xd 1

(the 1 means poll every second)... do this when the system is running "slow" Cut n paste maybe 10 of em.

newlinuxnewbie · 09-17-2006, 07:33 PM

Load is low now, right under 3, but iowait is still very high.

iostat
avg-cpu: %user %nice %sys %iowait %idle
0.54 0.00 7.31 67.89 24.26

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
hdc 63.33 3242.88 110.20 1386762034 47124856
hdc1 0.00 0.00 0.00 312 0
hdc2 63.33 3242.87 110.20 1386761634 47124856
hda 111.61 4586.83 419.58 1961483814 179428302
hda1 0.00 0.02 0.00 6816 182
hda2 20.72 148.44 270.37 63477978 115617598
hda3 2.00 4.48 21.37 1913914 9140140
hda4 0.00 0.00 0.00 2 0
hda5 5.38 34.61 37.41 14802088 15997320
hda6 0.97 6.92 10.60 2961010 4533602
hda7 0.80 9.13 1.49 3905388 638678
hda8 81.73 4383.23 78.34 1874415738 33500782

iostat -xd 1
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
hdc 389.00 5.00 73.00 2.00 3680.00 56.00 1840.00 28.00 49.81 1.68 22.67 7.87 59.00
hdc1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hdc2 389.00 5.00 73.00 2.00 3680.00 56.00 1840.00 28.00 49.81 1.68 22.67 7.87 59.00
hda 182.00 12.00 27.00 8.00 1672.00 112.00 836.00 56.00 50.97 0.55 15.71 6.86 24.00
hda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda3 0.00 2.00 0.00 2.00 0.00 32.00 0.00 16.00 16.00 0.00 0.00 0.00 0.00
hda4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda6 0.00 3.00 0.00 2.00 0.00 40.00 0.00 20.00 20.00 0.00 0.00 0.00 0.00
hda7 0.00 6.00 0.00 2.00 0.00 16.00 0.00 8.00 8.00 0.00 0.00 0.00 0.00
hda8 182.00 1.00 27.00 2.00 1672.00 24.00 836.00 12.00 58.48 0.55 18.97 8.28 24.00

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
hdc 488.00 0.00 72.00 0.00 4488.00 0.00 2244.00 0.00 62.33 1.62 22.36 9.44 68.00
hdc1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hdc2 488.00 0.00 72.00 0.00 4488.00 0.00 2244.00 0.00 62.33 1.62 22.36 9.44 68.00
hda 182.00 0.00 26.00 0.00 1664.00 0.00 832.00 0.00 64.00 0.34 13.08 10.00 26.00
hda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda8 182.00 0.00 26.00 0.00 1664.00 0.00 832.00 0.00 64.00 0.34 13.08 10.00 26.00

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
hdc 476.00 0.00 67.00 0.00 4376.00 0.00 2188.00 0.00 65.31 0.97 13.28 7.46 50.00
hdc1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hdc2 476.00 0.00 67.00 0.00 4376.00 0.00 2188.00 0.00 65.31 0.97 13.28 7.46 50.00
hda 189.00 0.00 30.00 0.00 1752.00 0.00 876.00 0.00 58.40 0.36 12.00 9.33 28.00
hda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda8 189.00 0.00 30.00 0.00 1752.00 0.00 876.00 0.00 58.40 0.36 12.00 9.33 28.00

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
hdc 447.00 0.00 74.00 0.00 4144.00 0.00 2072.00 0.00 56.00 0.95 13.65 7.30 54.00
hdc1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hdc2 447.00 0.00 74.00 0.00 4144.00 0.00 2072.00 0.00 56.00 0.95 13.65 7.30 54.00
hda 182.00 0.00 25.00 0.00 1664.00 0.00 832.00 0.00 66.56 0.20 7.20 8.00 20.00
hda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda8 182.00 0.00 25.00 0.00 1664.00 0.00 832.00 0.00 66.56 0.20 7.20 8.00 20.00

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
hdc 492.00 0.00 66.00 0.00 4504.00 0.00 2252.00 0.00 68.24 0.83 10.45 6.67 44.00
hdc1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hdc2 492.00 0.00 66.00 0.00 4504.00 0.00 2252.00 0.00 68.24 0.83 10.45 6.67 44.00
hda 147.00 0.00 22.00 0.00 1344.00 0.00 672.00 0.00 61.09 0.12 6.36 4.55 10.00
hda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda8 147.00 0.00 22.00 0.00 1344.00 0.00 672.00 0.00 61.09 0.12 6.36 4.55 10.00

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
hdc 451.00 5.00 82.00 2.00 4216.00 56.00 2108.00 28.00 50.86 1.22 16.19 6.67 56.00
hdc1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hdc2 451.00 5.00 82.00 2.00 4216.00 56.00 2108.00 28.00 50.86 1.22 16.19 6.67 56.00
hda 202.00 3.00 30.00 2.00 1864.00 40.00 932.00 20.00 59.50 0.31 9.06 7.50 24.00
hda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda8 202.00 3.00 30.00 2.00 1864.00 40.00 932.00 20.00 59.50 0.31 9.06 7.50 24.00

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
hdc 421.00 0.00 66.00 0.00 3896.00 0.00 1948.00 0.00 59.03 4.37 66.36 8.18 54.00
hdc1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hdc2 421.00 0.00 66.00 0.00 3896.00 0.00 1948.00 0.00 59.03 4.37 66.36 8.18 54.00
hda 175.00 0.00 27.00 0.00 1608.00 0.00 804.00 0.00 59.56 0.38 14.81 7.04 19.00
hda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda8 175.00 0.00 27.00 0.00 1608.00 0.00 804.00 0.00 59.56 0.38 14.81 7.04 19.00

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
hdc 471.00 0.00 70.00 0.00 4344.00 0.00 2172.00 0.00 62.06 1.37 19.29 8.86 62.00
hdc1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hdc2 471.00 0.00 70.00 0.00 4344.00 0.00 2172.00 0.00 62.06 1.37 19.29 8.86 62.00
hda 189.00 0.00 27.00 0.00 1728.00 0.00 864.00 0.00 64.00 0.31 11.48 8.52 23.00
hda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda8 189.00 0.00 27.00 0.00 1728.00 0.00 864.00 0.00 64.00 0.31 11.48 8.52 23.00

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
hdc 457.00 0.00 81.00 0.00 4288.00 0.00 2144.00 0.00 52.94 1.49 18.77 7.90 64.00
hdc1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hdc2 457.00 0.00 81.00 0.00 4288.00 0.00 2144.00 0.00 52.94 1.49 18.77 7.90 64.00
hda 305.00 0.00 44.00 0.00 2800.00 0.00 1400.00 0.00 63.64 0.42 9.32 7.27 32.00
hda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda3 4.00 0.00 2.00 0.00 48.00 0.00 24.00 0.00 24.00 0.01 5.00 5.00 1.00
hda4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda8 301.00 0.00 42.00 0.00 2752.00 0.00 1376.00 0.00 65.52 0.41 9.52 7.62 32.00

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
hdc 453.00 0.00 65.00 0.00 4160.00 0.00 2080.00 0.00 64.00 0.77 11.54 6.92 45.00
hdc1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hdc2 453.00 0.00 65.00 0.00 4160.00 0.00 2080.00 0.00 64.00 0.77 11.54 6.92 45.00
hda 201.00 0.00 30.00 0.00 1840.00 0.00 920.00 0.00 61.33 0.23 8.00 6.67 20.00
hda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
hda8 201.00 0.00 30.00 0.00 1840.00 0.00 920.00 0.00 61.33 0.23 8.00 6.67 20.00

generic_user · 09-17-2006, 07:47 PM

You'll note that HD utilization seems to be about equal to IOwait... This isn't a coincidence.

I assume hdc is the drive with the data you're working with?

newlinuxnewbie · 09-17-2006, 07:54 PM

Data is on both hdc and hda. Is there a way to reduce the iowait?

generic_user · 09-17-2006, 07:57 PM

It's a limitation of your drive(s). Not much you can do about that, lease you do something like use squid as a transparent proxy, or, perhaps, add more ram to help with the file cache.

Raid 0, 1, or 10 would help as far as reads go (which looks like most of your traffic).

newlinuxnewbie · 09-18-2006, 09:19 AM

Load is going crazy again, cpu, ram, and iowait is very low.
Can someone tell me why the load is so high?

top - 09:10:12 up 3 days, 12:36, 2 users, load average: 154.25, 156.11, 152.1
Tasks: 540 total, 1 running, 539 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.7% us, 3.0% sy, 0.0% ni, 0.0% id, 92.1% wa, 3.3% hi, 0.0% si
Mem: 970772k total, 456004k used, 514768k free, 2140k buffers
Swap: 2096472k total, 284308k used, 1812164k free, 84540k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14695 nobody 15 0 16000 5692 3484 D 0.0 0.6 0:01.05 httpd
15309 nobody 15 0 15320 5476 3272 D 0.0 0.6 0:00.57 httpd
2103 mysql 16 0 75260 4656 900 S 0.0 0.5 0:11.01 mysqld
2119 mysql 16 0 75260 4656 900 S 0.0 0.5 0:00.03 mysqld
2120 mysql 16 0 75260 4656 900 S 0.0 0.5 0:02.04 mysqld
2562 mysql 16 0 75260 4656 900 S 0.0 0.5 0:23.93 mysqld
2563 mysql 16 0 75260 4656 900 S 0.0 0.5 0:24.98 mysqld
2565 mysql 16 0 75260 4656 900 S 0.0 0.5 0:23.37 mysqld
2580 mysql 16 0 75260 4656 900 S 0.0 0.5 0:23.17 mysqld
2581 mysql 16 0 75260 4656 900 S 0.0 0.5 0:22.87 mysqld
2582 mysql 15 0 75260 4656 900 S 0.0 0.5 0:25.20 mysqld
2583 mysql 16 0 75260 4656 900 S 0.0 0.5 0:23.72 mysqld
2585 mysql 16 0 75260 4656 900 S 0.0 0.5 0:25.31 mysqld
2603 mysql 16 0 75260 4656 900 S 0.0 0.5 0:24.49 mysqld
2650 mysql 16 0 75260 4656 900 S 0.0 0.5 0:22.22 mysqld
2677 mysql 16 0 75260 4656 900 S 0.0 0.5 0:23.99 mysqld
2679 mysql 16 0 75260 4656 900 S 0.0 0.5 0:23.59 mysqld

avg-cpu: %user %nice %sys %iowait %idle
1.76 0.00 6.24 8.92 83.08

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 9.86 691.10 228.49 210564592 69617124
sda1 0.00 0.00 0.00 1334 4
sda2 32.10 679.87 206.55 207145154 62933224
sda3 4.15 11.22 21.94 3417249 6683896
sdb 27.05 3884.42 154.88 1183511452 47187992
sdb1 46.09 3884.41 154.88 1183510156 47187992

generic_user · 09-18-2006, 11:19 AM

Quote:

Originally Posted by newlinuxnewbie

Load is going crazy again, cpu, ram, and iowait is very low.
Can someone tell me why the load is so high?

top - 09:10:12 up 3 days, 12:36, 2 users, load average: 154.25, 156.11, 152.1
Tasks: 540 total, 1 running, 539 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.7% us, 3.0% sy, 0.0% ni, 0.0% id, 92.1% wa, 3.3% hi, 0.0% si
Mem: 970772k total, 456004k used, 514768k free, 2140k buffers
Swap: 2096472k total, 284308k used, 1812164k free, 84540k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14695 nobody 15 0 16000 5692 3484 D 0.0 0.6 0:01.05 httpd
15309 nobody 15 0 15320 5476 3272 D 0.0 0.6 0:00.57 httpd
2103 mysql 16 0 75260 4656 900 S 0.0 0.5 0:11.01 mysqld
2119 mysql 16 0 75260 4656 900 S 0.0 0.5 0:00.03 mysqld
2120 mysql 16 0 75260 4656 900 S 0.0 0.5 0:02.04 mysqld
2562 mysql 16 0 75260 4656 900 S 0.0 0.5 0:23.93 mysqld
2563 mysql 16 0 75260 4656 900 S 0.0 0.5 0:24.98 mysqld
2565 mysql 16 0 75260 4656 900 S 0.0 0.5 0:23.37 mysqld
2580 mysql 16 0 75260 4656 900 S 0.0 0.5 0:23.17 mysqld
2581 mysql 16 0 75260 4656 900 S 0.0 0.5 0:22.87 mysqld
2582 mysql 15 0 75260 4656 900 S 0.0 0.5 0:25.20 mysqld
2583 mysql 16 0 75260 4656 900 S 0.0 0.5 0:23.72 mysqld
2585 mysql 16 0 75260 4656 900 S 0.0 0.5 0:25.31 mysqld
2603 mysql 16 0 75260 4656 900 S 0.0 0.5 0:24.49 mysqld
2650 mysql 16 0 75260 4656 900 S 0.0 0.5 0:22.22 mysqld
2677 mysql 16 0 75260 4656 900 S 0.0 0.5 0:23.99 mysqld
2679 mysql 16 0 75260 4656 900 S 0.0 0.5 0:23.59 mysqld

avg-cpu: %user %nice %sys %iowait %idle
1.76 0.00 6.24 8.92 83.08

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 9.86 691.10 228.49 210564592 69617124
sda1 0.00 0.00 0.00 1334 4
sda2 32.10 679.87 206.55 207145154 62933224
sda3 4.15 11.22 21.94 3417249 6683896
sdb 27.05 3884.42 154.88 1183511452 47187992
sdb1 46.09 3884.41 154.88 1183510156 47187992

You're still not including the most important iostat -xd parameter!

Looks like you're using a lot of swap space too... and there are a LOT of mysqld processes... Alll using ram, and all in the sleep state.

newlinuxnewbie · 09-22-2006, 01:35 PM

Quote:

Originally Posted by generic_user

You're still not including the most important iostat -xd parameter!

Looks like you're using a lot of swap space too... and there are a LOT of mysqld processes... Alll using ram, and all in the sleep state.

Load is about to go crazy again. Please help me.

iostat -xd

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 5.15 90.24 13.26 9.12 1296.53 794.82 648.26 397.41 93.48 0.21 9.47 2.74 6.12
sdb 0.49 60.94 33.41 0.94 3832.42 495.02 1916.21 247.51 125.97 3.93 113.53 11.71 40.22

iostat

avg-cpu: %user %nice %sys %iowait %idle
0.74 0.00 4.41 43.56 51.30

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 22.37 1296.44 794.76 328764637 201543044
sda1 0.00 0.01 0.00 1702 30
sda2 7.21 764.63 5.21 193902267 1320314
sda3 95.32 450.29 727.29 114188556 184433088
sda4 0.00 0.00 0.00 2 0
sda5 2.71 2.29 19.22 581475 4873540
sda6 1.08 16.11 4.04 4085082 1024536
sda7 10.59 48.72 36.00 12355496 9129584
sda8 0.85 14.39 3.00 3649058 761952
sdb 34.36 3832.66 494.98 971922964 125522202
sdb1 95.78 3832.65 494.98 971921668 125522202

top - 13:38:02 up 2 days, 22:24, 3 users, load average: 30.47, 30.46, 50.84
Tasks: 262 total, 2 running, 260 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.7% us, 5.0% sy, 0.0% ni, 0.0% id, 94.4% wa, 0.0% hi, 0.0% si
Mem: 1003172k total, 1000736k used, 2436k free, 2312k buffers
Swap: 2048248k total, 52076k used, 1996172k free, 933428k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19226 nobody 15 0 360m 47m 46m S 0.0 4.8 0:02.31 httpd
20165 nobody 15 0 341m 32m 31m S 0.0 3.3 0:00.38 httpd
19885 nobody 15 0 201m 29m 28m S 0.3 3.0 0:00.73 httpd
19983 nobody 15 0 71672 28m 27m S 0.0 2.9 0:00.28 httpd
19915 nobody 16 0 71612 28m 27m S 0.0 2.9 0:00.35 httpd
20105 nobody 15 0 108m 28m 27m S 0.0 2.9 0:00.52 httpd
19458 nobody 15 0 357m 27m 26m S 0.0 2.8 0:00.49 httpd
19484 nobody 15 0 202m 25m 24m S 0.0 2.6 0:00.67 httpd
20007 nobody 15 0 80544 24m 23m S 0.0 2.5 0:00.42 httpd
19880 nobody 15 0 202m 23m 22m D 0.3 2.4 0:00.59 httpd
20209 nobody 15 0 201m 22m 21m D 0.0 2.3 0:00.16 httpd
19480 nobody 15 0 89552 22m 21m D 0.0 2.3 0:00.95 httpd
19875 nobody 15 0 162m 21m 20m D 0.0 2.2 0:00.57 httpd
19361 nobody 15 0 162m 21m 20m S 0.0 2.2 0:00.85 httpd
19506 nobody 15 0 165m 21m 20m D 0.0 2.1 0:00.68 httpd
19913 nobody 15 0 71612 20m 19m S 0.0 2.1 0:00.23 httpd
19997 nobody 15 0 461m 20m 19m S 0.3 2.1 0:00.53 httpd
19851 nobody 15 0 157m 20m 19m S 0.0 2.1 0:00.83 httpd
20207 nobody 15 0 202m 20m 18m D 0.0 2.1 0:00.15 httpd
20224 nobody 15 0 33084 20m 19m D 0.3 2.1 0:00.13 httpd
20164 nobody 15 0 225m 19m 18m S 0.0 2.0 0:00.14 httpd
20132 nobody 15 0 106m 19m 18m D 0.3 2.0 0:00.32 httpd
20200 nobody 15 0 71616 18m 17m D 0.0 1.9 0:00.12 httpd
20191 nobody 15 0 71524 17m 17m D 0.0 1.8 0:00.13 httpd
19344 nobody 15 0 85648 17m 16m S 0.0 1.8 0:00.63 httpd
19947 nobody 15 0 225m 17m 16m S 0.0 1.8 0:00.41 httpd
19911 nobody 15 0 225m 17m 16m S 0.0 1.8 0:00.33 httpd
19409 nobody 16 0 71524 17m 16m S 0.0 1.7 0:00.21 httpd
19427 nobody 15 0 80600 16m 15m S 0.0 1.7 0:00.37 httpd
19356 nobody 15 0 201m 16m 15m S 0.3 1.7 0:00.86 httpd
19198 nobody 15 0 88932 16m 15m S 0.0 1.7 0:00.67 httpd
19485 nobody 15 0 225m 16m 15m D 0.0 1.7 0:00.60 httpd
19354 nobody 15 0 225m 16m 15m S 0.0 1.6 0:00.73 httpd
19182 nobody 15 0 225m 16m 15m S 0.0 1.6 0:00.65 httpd
19487 nobody 15 0 225m 16m 15m D 0.0 1.6 0:00.55 httpd
20211 nobody 15 0 202m 16m 14m D 0.3 1.6 0:00.10 httpd
19483 nobody 15 0 201m 15m 15m S 0.0 1.6 0:00.72 httpd
19376 nobody 15 0 89660 15m 14m S 0.3 1.6 0:00.71 httpd