high cpu usage on %us and %sy

melano · 10-17-2012, 10:47 PM

top - 10:44:05 up 4 days, 2:46, 2 users, load average: 52.64, 58.96, 63.57
Tasks: 1399 total, 59 running, 1340 sleeping, 0 stopped, 0 zombie
Cpu(s): 27.7% us, 58.8% sy, 0.0% ni, 11.1% id, 2.0% wa, 0.0% hi, 0.5% si
Mem: 264247064k total, 263576596k used, 670468k free, 517672k buffers
Swap: 24579432k total, 238068k used, 24341364k free, 162072660k cached

is there a problem with running process?

druuna · 10-18-2012, 01:04 AM

With the information posted (one a very small time frame) it is hard to make any definite conclusion. With that in mind:

I do see a possible problem, but that is not with the CPU load:

Code:

top - 10:44:05 up 4 days, 2:46, 2 users, load average: 52.64, 58.96, 63.57

Those numbers are very high. The system seems to have a problem reading and/or writing from the disk(s). This might be due to slow disk(s) and/or multiple processes trying to access the disk(s) at the same time.

syg00 · 10-18-2012, 03:58 AM

The %wa looks o.k for that high a loadavg - I don't think disk(s) are the problem. More likely some crappy software parking threads in uninterrupible sleep and forgetting about them. HTTP and Oracle come to mind. Run the following and post the output

Code:

top -b -n 1 | awk '{if (NR <=7) print; else if ($8 == "D") {print; count++} } END {print "Total status D: "count}' > topsave.txt

melano · 10-18-2012, 04:23 AM

The OS was hang and restarted by itself because high load usage. I'm using Red Hat Enterprise Linux AS release 4 (Nahant Update 8) x86_64 and running Oracle DB 10g Enterprise Edition Release 10.2.0.3.0 - 64bit.

Can someone help me? where is the problem?

chrism01 · 10-18-2012, 04:34 AM

We need more detail eg show the complete output from top ie including the process list & also the output from syg00's cmd.

melano · 10-18-2012, 04:51 AM

after restart.

top - 16:48:14 up 2:07, 2 users, load average: 19.32, 22.87, 23.47
Tasks: 1187 total, 7 running, 1180 sleeping, 0 stopped, 0 zombie
Cpu(s): 10.2% us, 0.9% sy, 0.0% ni, 81.5% id, 7.2% wa, 0.0% hi, 0.2% si
Mem: 264247064k total, 214899436k used, 49347628k free, 2379396k buffers
Swap: 24579432k total, 0k used, 24579432k free, 161129136k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10772 oracle 17 0 160g 3.5g 3.5g R 98.0 1.4 59:30.72 oracle
22463 oracle 16 0 160g 7.8g 7.8g R 95.0 3.1 9:22.64 oracle
7450 oracle 16 0 160g 6.8g 6.7g S 84.1 2.7 0:45.21 oracle
10107 oracle 16 0 160g 14g 14g R 82.1 5.8 25:38.92 oracle
4797 oracle 17 0 160g 383m 375m R 68.6 0.1 0:14.54 oracle
15322 oracle 17 0 160g 4.3g 4.3g R 56.1 1.7 1:42.66 oracle
3508 oracle 16 0 160g 19g 19g R 41.6 7.6 30:42.75 oracle
23651 oracle 16 0 160g 6.0g 6.0g D 31.0 2.4 1:16.01 oracle
30281 oracle 16 0 160g 3.4g 3.4g S 22.8 1.3 0:21.84 oracle
2290 oracle 15 0 160g 25g 25g D 9.2 10.1 6:24.66 oracle
20227 oracle 16 0 160g 371m 342m S 9.2 0.1 0:02.65 oracle
8573 oracle 16 0 161g 30g 28g D 7.6 12.1 6:12.21 oracle
22317 oracle 16 0 160g 376m 361m S 5.6 0.1 0:01.04 oracle
3357 oracle 16 0 160g 8.3g 8.2g D 4.9 3.3 0:34.45 oracle
3572 oracle 16 0 160g 46m 38m S 4.3 0.0 4:02.37 oracle
11073 oracle 16 0 160g 487m 468m S 4.3 0.2 0:03.61 oracle
26054 oracle 15 0 160g 35g 35g D 3.0 14.0 4:13.07 oracle

==================================================================================================== =======

the output top -b -n 1 | awk '{if (NR <=7) print; else if ($8 == "D") {print; count++} } END {print "Total status D: "count}' > topsave.txt

top - 16:49:40 up 2:08, 2 users, load average: 17.84, 21.38, 22.88
Tasks: 1180 total, 11 running, 1169 sleeping, 0 stopped, 0 zombie
Cpu(s): 13.2% us, 1.6% sy, 0.0% ni, 75.7% id, 9.3% wa, 0.0% hi, 0.2% si
Mem: 264247064k total, 214989028k used, 49258036k free, 2379940k buffers
Swap: 24579432k total, 0k used, 24579432k free, 161418584k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16207 oracle 15 0 160g 289m 260m D 79.2 0.1 0:17.19 oracle
16815 oracle 16 0 160g 1.5g 1.5g D 60.4 0.6 0:05.04 oracle
8573 oracle 16 0 161g 30g 29g D 28.3 12.2 6:21.34 oracle
2290 oracle 15 0 160g 25g 25g D 17.0 10.3 6:35.37 oracle
6482 oracle 15 0 160g 54m 29m D 1.9 0.0 1:36.39 oracle
Total status D: 5

syg00 · 10-20-2012, 04:12 AM

As suspected, has to be Oracle - looking at those numbers, it is just starting too many processes. Starting to see some %wa, so @druuna might have been right - hard to say definitely from that minimal data. I wonder if the status D count goes up as the loadavg gets bigger.

I have no idea how you control/limit Oracle.

melano · 10-21-2012, 08:40 PM

Quote:

Originally Posted by syg00

As suspected, has to be Oracle - looking at those numbers, it is just starting too many processes. Starting to see some %wa, so @druuna might have been right - hard to say definitely from that minimal data. I wonder if the status D count goes up as the loadavg gets bigger.

I have no idea how you control/limit Oracle.

how do i limit oracle process?