Constantly high CPU usage

andka16 · 10-10-2008, 07:14 AM

I've a RHEL4 server running in vmware ESX 3.5.
The server has 5.5GB RAM and 1 CPU
Oracle with 2 db instances are also installed.

The cpu is running at 100% almost all the time.
I've tried to see exactly what process/program is using all that cpu but im not able to see anything strange while using ex. top

Someone who could help regarding this problem?

rhelaine · 10-10-2008, 04:09 PM

what is the server hardware? i suppose it is not a single core CPU running ESX with 2 oracle DB...

paulsm4 · 10-11-2008, 02:00 AM

Hi -

We need a few more details to help you troubleshoot.

Q: What exactly is telling you that your "CPU is running at 100%"? iostat? vmstat? sar? Something else?

Please cut/paste sample output.

Q: Do you notice the system appear to "hang" (one should expect this if the CPU were indeed max'ed out)?

Q: When this occurs, and you run "top" and sort by CPU utilization, which process appears to be consuming the most CPU?

Please cut/paste sample output of this, too.

ghodkiller · 10-11-2008, 10:07 AM

do you use a statistics program for oracle ? what do they say ?

Debian

paulsm4 · 10-12-2008, 12:06 PM

Hi -

It doesn't look like andka16 ever posted before, and I'm not sure he's going to follow up with this post. But if he does, we definitely need more specific information to go on. Starting with "the exact message that led him to believe he's running 100% CPU". Who knows - maybe he's reading something in ESX (not his Red Hat VM at all)...

andka16 · 10-13-2008, 12:37 AM

Hi!
Sorry for the late response.

The server hardware is a Dell PowerEdge 6850
32GB RAM
4 x DualCore CPU

If i use the top-command it tells me that the server is using all it's CPU resources. 5-10% user 90-95% System

If i look inside the VI client on the performance it's also telling me it's using all it's resources.

The ESX itself is running smoothly, it's just this vm that seem to use up every resource it can get.

The users of this server experiencing slow responses almost all the time.

When running the top command i can't see the process that causing the high cpu load.

I haven't used any statistics program on oracle...

Thank you!
/Andreas

andka16 · 10-13-2008, 12:41 AM

Output of top-command....

top - 07:40:16 up 6 days, 20:05, 1 user, load average: 5.07, 5.28, 4.73
Tasks: 353 total, 8 running, 344 sleeping, 0 stopped, 1 zombie
Cpu(s): 11.8% us, 87.8% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.3% hi, 0.0% si
Mem: 5319836k total, 5298208k used, 21628k free, 78228k buffers
Swap: 2031608k total, 490928k used, 1540680k free, 3728556k cached

PID USER PR NI %CPU TIME+ %MEM VIRT RES SHR S COMMAND
30509 oracle 24 0 17.8 1:42.50 0.9 1032m 47m 46m R oracle
30805 daby 15 0 8.6 0:00.26 0.2 27424 11m 8488 S todayx
27930 blodc152 15 0 7.9 0:49.82 0.2 27356 11m 8396 S todays
9358 blodc152 17 0 6.9 147:23.65 7.6 407m 394m 9740 S todayx
30842 oracle 21 0 6.3 0:00.19 0.3 1032m 15m 13m R oracle
21338 blodc145 17 0 5.9 87:00.30 4.7 255m 241m 9668 S todayx
25276 root 17 0 1.3 0:01.54 0.0 2252 1184 788 R top
8583 daby 19 0 0.7 38:32.03 3.4 836m 175m 32m S java
30313 daby 17 0 0.7 0:00.07 0.0 5444 1096 944 S sh
30711 daby 15 0 0.7 0:00.02 0.0 2444 556 448 S profile
10667 daby 25 0 0.3 37:50.51 3.3 827m 171m 33m S java
24028 root 16 0 0.3 0:00.28 0.0 7200 2312 1852 S sshd
30363 daby 15 0 0.3 0:00.07 0.0 3012 556 448 S profile
30713 daby 19 0 0.3 0:00.01 0.0 5360 608 524 S grep
1 root 16 0 0.0 0:07.46 0.0 3308 592 504 S init
2 root RT 0 0.0 0:00.00 0.0 0 0 0 S migration/0
3 root 34 19 0.0 0:00.29 0.0 0 0 0 S ksoftirqd/0

andka16 · 10-13-2008, 12:43 AM

Output from iostat-command

Device: tps | Blk_read/s | Blk_wrtn/s | Blk_read | Blk_wrtn
sda 2,56| 17,73 | 86,61 | 10478212 | 51188660
sda1 0,00| 0,03 | 0,00 | 15778 | 100
sda2 11,81| 17,70 | 86,61 | 10461298 | 51188560
sdb 3,61| 81,87 | 57,21 | 48390994 | 33816408
sdc 0,22| 1,93 | 3,15 | 1142082 | 1862152
sdd 8,82| 1079,34 | 9,87 |637944498 | 5836256
sde 2,03| 11,39 | 47,88 | 6734074 | 28298176
sdf 0,53| 35,93 | 5,83 | 21234002 | 3443784
sdg 6,12| 757,66 | 785,38 |447813666 |464200912
sdh 0,01| 0,07 | 0,01 | 41826 | 5584
sdi 4,05| 99,61 | 60,67 | 58876418 | 35856312
dm-0 10,77| 14,33 | 81,67 | 8469554 | 48268552
dm-1 1,04| 3,37 | 4,37 | 1990176 | 2920008
dm-2 9,59| 99,61 | 60,67 | 58872642 | 35856312
dm-3 0,01| 0,06 | 0,01 | 38034 | 5584
dm-4 101,46| 757,65 | 785,38 |447809874 |464200912
dm-5 0,89| 35,92 | 5,83 | 21230210 | 3443784
dm-6 6,11| 11,39 | 47,88 | 6729786 | 28298176
dm-7 9,54| 1079,33 | 9,87 |637940706 | 5836256
dm-8 0,47| 1,93 | 3,15 | 1138290 | 1862152
dm-9 8,82| 81,87 | 57,21 | 48387202 | 33816408

andka16 · 10-13-2008, 12:59 AM

iostat -c 2 4

avg-cpu: %user %nice %sys %iowait %idle
12,00 0,00 88,00 0,00 0,00

avg-cpu: %user %nice %sys %iowait %idle
10,50 0,00 89,50 0,00 0,00

avg-cpu: %user %nice %sys %iowait %idle
11,44 0,00 88,56 0,00 0,00

avg-cpu: %user %nice %sys %iowait %idle
11,44 0,00 88,56 0,00 0,00

syg00 · 10-13-2008, 02:52 AM

How many of those CPUs are you actually using ???. When in top, hit the "1" (numeric one) key.

andka16 · 10-13-2008, 03:57 AM

Quote:

Originally Posted by syg00

How many of those CPUs are you actually using ???. When in top, hit the "1" (numeric one) key.

Im using 1 vCpu.
We installed this machine as the supplier suggested.
After some research i found out that if we have to add another vCpu to this machine we have to reinstall the OS

is this true???

syg00 · 10-13-2008, 06:47 AM

oops, you did say that in the initial post.
Depending on the distro/release, SMP may be supported - these days, it seems to be the default. Never understood why it wasn't all the time.

However: - if your supplier insisted on a uniprocessor implementation, that could (would) imply their code is not multiprocessor safe. Talk to them before turning more engine(s) on.

andka16 · 10-13-2008, 07:02 AM

We are running.....

Red Hat Enterprise Linux AS Release 4 (Nahant Update 6)
Kernel 2.6.9-67.0.7.ELsmp on an i686

andka16 · 10-13-2008, 08:01 AM

It's easy to add another CPU to a server running inside Vmware ESX although i've heard somewhere that if you installed it using single CPU you have to reinstall the OS to make it support more than one CPU.
Anyone who knows where to find more information about this?

paulsm4 · 10-13-2008, 08:41 AM

Yup- you're definitely CPU bound, and you're definitely only using one of two CPUs.

I would create a new VMWare image, installing Redhat explicitly allowing SMP support...

Code:

PID   USER     PR NI %CPU     TIME+ %MEM  VIRT  RES   SHR S COMMAND
30509 oracle   24 0  17.8   1:42.50  0.9 1032m  47m   46m R oracle
30805 daby     15 0   8.6   0:00.26  0.2 27424  11m  8488 S todayx
27930 blodc152 15 0   7.9   0:49.82  0.2 27356  11m  8396 S todays
9358  blodc152 17 0   6.9 147:23.65  7.6  407m 394m  9740 S todayx
30842 oracle   21 0   6.3   0:00.19  0.3 1032m  15m   13m R oracle
21338 blodc145 17 0   5.9  87:00.30  4.7  255m 241m  9668 S todayx
25276 root     17 0   1.3   0:01.54  0.0  2252 1184   788 R top
8583  daby     19 0   0.7  38:32.03  3.4  836m 175m   32m S java
30313 daby     17 0   0.7   0:00.07  0.0  5444 1096   944 S sh
30711 daby     15 0   0.7   0:00.02  0.0  2444  556   448 S profile
10667 daby     25 0   0.3  37:50.51  3.3  827m 171m   33m S java
24028 root     16 0   0.3   0:00.28  0.0  7200 2312  1852 S sshd
30363 daby     15 0   0.3   0:00.07  0.0  3012  556   448 S profile
30713 daby     19 0   0.3   0:00.01  0.0  5360  608   524 S grep
1     root     16 0   0.0   0:07.46  0.0  3308  592   504 S init
2     root     RT 0   0.0   0:00.00  0.0     0    0     0 S migration/0
3     root     34 19  0.0   0:00.29  0.0     0    0     0 S ksoftirqd/0

Code:

iostat -c 2 4                           
                                       
avg-cpu: %user  %nice  %sys %iowait %idle
         12,00   0,00 88,00    0,00  0,00             
                                       
avg-cpu: %user  %nice  %sys %iowait %idle
         10,50   0,0  89,50    0,00  0,00             
                                       
avg-cpu: %user  %nice  %sys %iowait %idle
         11,44   0,00 88,56    0,00  0,00             
                                       
avg-cpu: %user  %nice  %sys %iowait %idle
         11,44   0,00 88,56    0,00  0,00