100% CPU utilization

linuxandtsm · 02-28-2012, 11:46 AM

Hi got below messages from oracle enterprise manager
how to check/solve what is causing this issue.

Code:

Target Name=EnterpriseManager0.server_name.com 
Target type=Oracle Application Server 
Host=server_name.com 
Occurred At=<Date and Time here>
Message=CPU Utilization is 100% 
Metric=CPU Usage (%) 
Metric value=100 
Severity=Critical 
Acknowledged=No 
Notification Rule Name=Application Server Availability and Critical States 
Notification Rule Owner=SYSMAN

Below are few command outputs

Code:

top command after hitting 1

top - 12:46:43 up 134 days, 21:56,  3 users,  load average: 0.05, 0.07, 0.08
Tasks: 240 total,   1 running, 223 sleeping,   0 stopped,  16 zombie
Cpu0  :  1.3%us,  2.0%sy,  0.0%ni, 96.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  :  4.7%us,  3.7%sy,  0.0%ni, 91.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  16189316k total, 16054464k used,   134852k free,   310880k buffers
Swap: 17391684k total,   216840k used, 17174844k free, 14139716k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 8241 oracle    16   0  366m  80m  19m S  1.0  0.5 156:45.68 emagent
 2110 root      15   0 10676  928  724 S  0.7  0.0 246:24.22 vmware-guestd
 8307 oracle    15   0  632m 478m 474m S  0.3  3.0  19:58.90 oracle
28488 oracle    15   0 12736 1200  816 S  0.3  0.0   0:01.09 top
    1 root      15   0 10344  680  572 S  0.0  0.0   1:50.28 init
    2 root      RT  -5     0    0    0 S  0.0  0.0   1:30.33 migration/0
    3 root      34  19     0    0    0 S  0.0  0.0   0:16.58 ksoftirqd/0
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/0
    5 root      RT  -5     0    0    0 S  0.0  0.0   1:25.42 migration/1
    6 root      34  19     0    0    0 S  0.0  0.0   0:20.81 ksoftirqd/1
    7 root      RT  -5     0    0    0 S  0.0  0.0   0:00.02 watchdog/1
    8 root      10  -5     0    0    0 S  0.0  0.0   0:12.31 events/0
    9 root      10  -5     0    0    0 S  0.0  0.0   0:03.39 events/1
   10 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 khelper
  139 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kthread
  144 root      10  -5     0    0    0 S  0.0  0.0   0:49.44 kblockd/0
  145 root      10  -5     0    0    0 S  0.0  0.0   0:12.54 kblockd/1
  146 root      14  -5     0    0    0 S  0.0  0.0   0:00.00 kacpid
  202 root      14  -5     0    0    0 S  0.0  0.0   0:00.00 cqueue/0
  203 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 cqueue/1
  206 root      14  -5     0    0    0 S  0.0  0.0   0:00.00 khubd
  208 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kseriod
  279 root      15   0     0    0    0 S  0.0  0.0   0:00.00 pdflush
  280 root      15   0     0    0    0 S  0.0  0.0   0:29.99 pdflush
  281 root      10  -5     0    0    0 S  0.0  0.0  10:56.72 kswapd0
  282 root      14  -5     0    0    0 S  0.0  0.0   0:00.00 aio/0
  283 root      14  -5     0    0    0 S  0.0  0.0   0:00.00 aio/1
  424 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 kpsmoused
  471 root      10  -5     0    0    0 S  0.0  0.0   0:00.27 mpt_poll_0
  472 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_0
  476 root      16  -5     0    0    0 S  0.0  0.0   0:00.00 ata/0
  477 root      16  -5     0    0    0 S  0.0  0.0   0:00.00 ata/1
  478 root      16  -5     0    0    0 S  0.0  0.0   0:00.00 ata_aux
  485 root      18  -5     0    0    0 S  0.0  0.0   0:00.00 kstriped
  498 root      20  -5     0    0    0 S  0.0  0.0   0:00.00 ksnapd
  513 root      10  -5     0    0    0 S  0.0  0.0   0:36.30 kjournald
  540 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 kauditd
  574 root      20  -4 12688  784  392 S  0.0  0.0   0:00.34 udevd

Code:

[root@server_name ~]# iostat
Linux 2.6.18-128.1.14.el5 (server_name)  02/28/2012

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.09    0.00    1.28    0.63    0.00   95.00

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              17.33       388.11       364.71 4524096378 4251327378
sda1              0.00         0.00         0.00       2794          4
sda2             17.33       388.11       364.71 4524088984 4251327374
dm-0              4.08         0.08         8.14     927876   94895678
dm-1              0.00         0.00         0.00      18626      31560
dm-2              0.01         0.03         0.03     405146     395240
dm-3              1.35         0.00        10.83       5410  126202472
dm-4              0.97         0.56         7.43    6582642   86625040
dm-5              1.18         0.37         9.38    4270722  109312456
dm-6              0.23         0.93         0.88   10843640   10258064
dm-7             44.69       386.08       328.02 4500418058 3823606792
dm-8              0.01         0.05         0.00     616498         72

Thanks in advance!

T3RM1NVT0R · 02-28-2012, 12:52 PM

Hi linuxandtsm,

Things that I can understand from the output:

1. It is RHEL5 machine.
2. It is up from quite a long. (134 days)
3. Are you running VMware on this machine? Because it appears that you are.
4. If you are running VMware on this machine then how many guests are configured?
5. Does Oracle application manager is configured to generate alerts related to this machine? Just asking it is not a reason for high utilization though.
6. Was there a sudden hike in the CPU utilization or it slowly and gradually reached there?
7. Any changes made to the system?

linuxandtsm · 02-28-2012, 01:52 PM

Hi T3RM1NVT0R,

Yes this is a VM of RHEL5 and oracle manager was configured to send alerts (like the one i posted).
How can i check if there was a sudden hike in CPU utilization ? and there was no chages/updates on ths VM.

PS: i also have a query on understanding %cpu column in top commands output. If it shows 1.0, what does it mean. Does it mean 100% cpu utilized ?

T3RM1NVT0R · 02-28-2012, 02:06 PM

Quote:

How can i check if there was a sudden hike in CPU utilization ? and there was no chages/updates on ths VM.

You cannot tell until and unless you have script running that keep a tab of CPU usage pattern.

Quote:

PS: i also have a query on understanding %cpu column in top commands output. If it shows 1.0, what does it mean. Does it mean 100% cpu utilized ?

Nope. That means that CPU utilization is 1%.

It might be possible that CPU might have hit 100% for a moment and then came down to normal utilization. This cannot be considered as high utilization because some processess sometime for a fraction of second causes CPU to spike upto 100.

If I am not wrong your alerting system throws an alert immediately it see that CPU pegged at 100% without even bothering for how long it was there at 100%.

For me high utilization is a situation wherein CPU is stuck at a particular percentage. Doesn't matter if it is 60% because then I know that there is some processes which is continuoulsy using 60% of my CPU.

linuxandtsm · 02-28-2012, 02:18 PM

truely educating answer!
Thanks!

mikaeltristan · 05-31-2012, 01:18 AM

Hi,

I am new to the linux. I have a problem, CPU usage over 90% and my pc runs very slow. I use Ubuntu 8.04.

This is the result with command 'Top'

Tasks: 595 total, 62 running, 533 sleeping, 0 stopped, 0 zombie
Cpu(s): 96.0%us, 3.2%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.8%si, 0.0%st
Mem: 3628740k total, 3504892k used, 123848k free, 11736k buffers
Swap: 4883752k total, 39112k used, 4844640k free, 3200912k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5093 root 20 0 1928 656 296 R 4 0.0 3:59.68 gzip
5148 root 20 0 1928 660 296 R 4 0.0 3:56.75 gzip
5160 root 20 0 1928 660 296 R 4 0.0 3:55.58 gzip
1927 root 20 0 1928 664 296 R 3 0.0 5:53.92 gzip
1928 root 20 0 1928 664 296 R 3 0.0 5:54.36 gzip
1929 root 20 0 1928 664 296 R 3 0.0 5:53.78 gzip
2007 root 20 0 1928 660 296 R 3 0.0 5:41.70 gzip
2130 root 20 0 1928 668 296 R 3 0.0 5:22.45 gzip
2382 root 20 0 1928 664 296 R 3 0.0 5:00.29 gzip
4175 root 20 0 1928 664 296 R 3 0.0 4:35.46 gzip
4176 root 20 0 1928 660 296 R 3 0.0 4:34.85 gzip
4371 root 20 0 1928 660 296 R 3 0.0 4:24.54 gzip
4479 root 20 0 1928 660 296 R 3 0.0 4:17.98 gzip

Please help me to resolve this problem...

Thanks,
Tristan

EDDY1 · 05-31-2012, 01:38 AM

Quote:

Originally Posted by mikaeltristan

Hi,

I am new to the linux. I have a problem, CPU usage over 90% and my pc runs very slow. I use Ubuntu 8.04.
Please help me to resolve this problem...

Thanks,
Tristan

You will probably want to start a new thread to get the proper assistance that you require. Unless you are using server version your release may be unsupported
https://wiki.ubuntu.com/Releases