LinuxQuestions.org
Help answer threads with 0 replies.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
Reload this Page Unknown CPU usage - top and ps don't show the cause
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 10-11-2009, 02:13 PM   #1
gimpy530
Member
 
Registered: Oct 2007
Posts: 98

Rep: Reputation: 16
Unknown CPU usage - top and ps don't show the cause

So a few weeks ago a Linux server was having high CPU usage, but we weren't able to determine what program was causing it. The top and ps commands (and others) didn't show any programs using a huge amount of CPU resources, but %idle was at 0 and our monitoring software showed 100% usage. Yet no process was found by ps and top to explain it. The usage was caused by a DBA doing some work, but we never found out why we could not see the application causing the usage.

Now here I am several weeks later and I have the same problem, but this time it is on my Palm Pre. I upgraded to 1.2.1, and now it runs very slowly. A quick top in an SSH session shows:
Code:
top - 00:42:34 up  1:06,  2 users,  load average: 7.13, 7.42, 7.64
Tasks:  94 total,   8 running,  86 sleeping,   0 stopped,   0 zombie
Cpu(s): 77.9%us, 21.8%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:    245036k total,   231908k used,    13128k free,    15500k buffers
Swap:   131064k total,       84k used,   130980k free,    46760k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                     
 1019 root      20   0  2932 1492  656 S 11.7  0.6   6:54.13 dbus-daemon                                 
    1 root      20   0  1960 1012  696 R  6.5  0.4   3:50.56 upstart                                     
  922 root      20   0  2896 1276  964 S  5.2  0.5   3:28.35 pmsyslogd                                   
14879 root      20   0  4688 1672 1356 R  1.6  0.7   0:00.05 luna-helper                                 
14881 root      20   0  4692 1612 1348 R  1.3  0.7   0:00.04 luna-helper                                 
11161 root      20   0  2476 1160  900 R  1.0  0.5   0:01.73 top                                         
14883 root      20   0  4556  824  656 R  0.6  0.3   0:00.02 luna-helper                                 
 1270 root      19  -1  217m  87m  15m S  0.3 36.4   3:23.93 LunaSysMgr                                  
14885 root      20   0  4584  684  556 R  0.3  0.3   0:00.01 luna-helper                                 
14887 root      20   0  4584  684  556 R  0.3  0.3   0:00.01 luna-helper                                 
14893 root      20   0  2452 1024  812 S  0.3  0.4   0:01.04 dropbear                                    
    2 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 kthreadd                                    
    3 root      15  -5     0    0    0 S  0.0  0.0   0:00.18 ksoftirqd/0                                 
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.02 watchdog/0                                  
    5 root      15  -5     0    0    0 S  0.0  0.0   0:01.78 events/0                                    
    6 root      15  -5     0    0    0 S  0.0  0.0   0:00.03 khelper                                     
   98 root      15  -5     0    0    0 S  0.0  0.0   0:00.12 kblockd/0
Now, assuming I'm not just being stupid (which is always a possibility) the first two values on the CPU line show a total of 100% usage, yet no process shows that. The ps command shows a similar result.

So, what is causing the high CPU usage? Why doesn't top and ps show it?
Last edited by gimpy530; 10-11-2009 at 02:17 PM.
 
Old 10-11-2009, 10:33 PM   #2
paulsm4
LQ Guru
 
Registered: Mar 2004
Distribution: SusE 8.2
Posts: 5,863
Blog Entries: 1

Rep: Reputation: Disabled
I agree that the numbers don't "add up" ...

... but I get the strong impression:

a) the high load average (you've got "7"; any chronic wait queue values over "2" is suspect) is definitely indicative of performance problems

b) the high cpu utilization...
... coupled with the fact that no single program seems to be hogging the CPU...

c) and the high memory usage ...

that maybe there's a memory issue: that maybe the LunaSysMgr GUI is hogging most of your 256MB RAM...

... even though "idle", "I/O wait" and swap usage are (essentially) zero

... I suspect more memory might improve things.

I also suspect that whatever your DBA was doing was also memory (vs CPU) intensive, and perhaps there might have been a memory issue there, too.

IMHO .. PSM
Last edited by paulsm4; 10-11-2009 at 10:36 PM.
 
Old 10-12-2009, 05:42 PM   #3
gimpy530
Member
 
Registered: Oct 2007
Posts: 98

Original Poster
Rep: Reputation: 16
Memory causing high CPU? I don't get it, why would that happen?
 
Old 10-12-2009, 07:18 PM   #4
lutusp
Member
 
Registered: Sep 2009
Distribution: Fedora
Posts: 835

Rep: Reputation: 102Reputation: 102
Quote:
Originally Posted by gimpy530 View Post
Memory causing high CPU? I don't get it, why would that happen?
The reason low RAM causes high CPU loading is because the system spends a lot of time in swapping operations. This also makes processes take longer in terms of wall time, even when the CPU time statistics look good -- because the system has to wait on drive storage rather than RAM storage.

It is very common on a low-RAM system for the system to be very slow but with no serious CPU loading taking place, all because the system is constantly swapping to the drive instead of reading and writing RAM.

Must of that time is spent in hardware wait states, not CPU time, so it doesn't show up in the CPU statistics, but the machine is very slow.
 
Old 10-12-2009, 07:51 PM   #5
AlucardZero
Senior Member
 
Registered: May 2006
Location: USA
Distribution: Debian
Posts: 4,824

Rep: Reputation: 615Reputation: 615Reputation: 615Reputation: 615Reputation: 615Reputation: 615
84k swap used is trivial.

Still, watch the si and so columns of
Code:
vmstat 5
.
 
Old 10-13-2009, 06:45 AM   #6
GazL
LQ Veteran
 
Registered: May 2008
Posts: 6,897

Rep: Reputation: 5019Reputation: 5019Reputation: 5019Reputation: 5019Reputation: 5019Reputation: 5019Reputation: 5019Reputation: 5019Reputation: 5019Reputation: 5019Reputation: 5019
Those %cpu figures for upstart, dbus, and pmsyslogd look high to me. I'd expect them to be close to or around 0 on a healthy system.

I'd be inclined to have a look through your logfiles for any repeated messages, check if some daemon or other keeps failing and being constantly restarted by upstart.


P.S. Read the manpage for ps and look what %cpu actually means. It's not what you're expecting it to.
Last edited by GazL; 10-13-2009 at 06:48 AM.
 
Old 10-14-2009, 06:16 PM   #7
gimpy530
Member
 
Registered: Oct 2007
Posts: 98

Original Poster
Rep: Reputation: 16
I re-installed the OS on my Palm Pre to fix the problem on that, so I will look into that server tomorrow. Right now it is showing high iowait, so even though these have similar symptoms they seem to be different issues. When this started on the server, I don't recall seeing that high iowait though.
 
Old 10-14-2009, 10:40 PM   #8
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 10,659
Blog Entries: 4

Rep: Reputation: 3940Reputation: 3940Reputation: 3940Reputation: 3940Reputation: 3940Reputation: 3940Reputation: 3940Reputation: 3940Reputation: 3940Reputation: 3940Reputation: 3940
Is DMA working properly on the I/O controller?
 
Old 10-15-2009, 01:32 PM   #9
gimpy530
Member
 
Registered: Oct 2007
Posts: 98

Original Poster
Rep: Reputation: 16
Code:
vmstat 5

procs                      memory      swap          io     system         cpu
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy wa id
 0  2  13680  17508   3856 7231880    0    0     2    22   14    10 13  2 20  5
 0  2  13680  17520   3864 7231852    0    0  5859   868  987  2429 16  2 82  0
 2  1  13680  17368   3272 7232596    0    0  5538   288  946  2443  5  2 93  0
 0  2  13680  17388   3284 7232564    0    0  5088   271  869  2329  4  2 94  0
 2  2  13680  17548   3176 7232508    0    0  5158   251  882  2365  5  2 93  0
 0  2  13680  17476   3124 7232644    0    0  5261   162  897  2357  5  3 93  0
 0  2  13680  17468   3012 7232756    0    0  4920   349  849  2276  5  2 94  0
 1  2  13680  17492   3040 7232704    0    0  4513   263  778  2155  4  1 95  0
Code:
iostat -x 1


avg-cpu:  %user   %nice    %sys   %idle
          78.57    0.00   21.43    0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
/dev/sda     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
/dev/sda1    0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
/dev/sda2    0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
/dev/sda3    0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
/dev/sda5    0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
/dev/sda6    0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
/dev/sda7    0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
/dev/sda8    0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
/dev/sda9    0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
/dev/sda10   0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
/dev/sda11   0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
/dev/sda12   0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
/dev/sda13   0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
/dev/sda14   0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
/dev/sdb   6514.29  85.71 10442.86 85.71    0.00 1371.43     0.00   685.71     0.13   395.71    3.74   1.34 1414.29
/dev/sdb1  6514.29  85.71 10442.86 85.71    0.00 1371.43     0.00   685.71     0.13   395.71    3.74   1.34 1414.29
How do I check if DMA is working?
Last edited by gimpy530; 10-15-2009 at 01:50 PM.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
CPU usage from top jr8rdt Linux - Newbie 10 08-17-2011 06:46 AM
Why doesnt top or free show swap memory usage ? exceed1 Linux - General 15 08-03-2009 08:38 PM
rpcbind? What is it and how do why does it show up on top as CPU eater? MikeyCarter Linux - Software 2 05-20-2008 07:05 PM
Top CPU usage explanation penguinco Fedora 3 11-30-2004 01:01 PM
need top clone- to measure CPU usage pao Linux - Software 4 10-07-2004 09:46 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 10:47 PM.

Contact Us - Advertising Info - Rules - Privacy - LQ Merchandise - Donations - Contributing Member - LQ Sitemap -
Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration