Red Hat This forum is for the discussion of Red Hat Linux. |
Notices |
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
Are you new to LinuxQuestions.org? Visit the following links:
Site Howto |
Site FAQ |
Sitemap |
Register Now
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
|
 |
01-22-2008, 10:46 AM
|
#1
|
Member
Registered: Jun 2006
Posts: 132
Rep:
|
high load average, low cpu usage !
hi,
recently, one of my centOS 4.4 server got slow, and I found it has high load average, low cpu usage, I din't find any specific process eating cpu or memory, anyway to find what's going on with this server?
top - 11:43:30 up 399 days, 16:37, 4 users, load average: 7.25, 7.26, 7.27
Tasks: 116 total, 2 running, 113 sleeping, 0 stopped, 1 zombie
Cpu(s): 0.2% us, 0.7% sy, 0.0% ni, 97.0% id, 2.2% wa, 0.0% hi, 0.0% si
Mem: 1034728k total, 993380k used, 41348k free, 36172k buffers
Swap: 1204864k total, 696488k used, 508376k free, 261064k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7389 root 17 0 2952 960 744 R 0.7 0.1 0:01.09 top
1 root 15 0 2792 440 408 S 0.0 0.0 11:56.27 init
2 root RT 0 0 0 0 S 0.0 0.0 2:54.24 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 1:05.87 ksoftirqd/0
4 root RT 0 0 0 0 S 0.0 0.0 83:42.93 migration/1
5 root 34 19 0 0 0 S 0.0 0.0 1:07.21 ksoftirqd/1
6 root 5 -10 0 0 0 S 0.0 0.0 0:03.50 events/0
7 root 5 -10 0 0 0 S 0.0 0.0 0:01.87 events/1
8 root 5 -10 0 0 0 S 0.0 0.0 0:00.01 khelper
9 root 15 -10 0 0 0 S 0.0 0.0 0:00.00 kacpid
[root@localhost ~]# vmstat 1 10
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 696488 40636 36332 261424 0 0 0 1 0 0 4 1 93 2
0 0 696488 40636 36336 261420 0 0 0 12 1021 1221 0 0 99 1
0 0 696488 40780 36336 261420 0 0 0 0 1027 1234 0 1 100 0
0 1 696488 40780 36344 261412 0 0 0 24 1171 1304 1 0 97 3
0 0 696488 40780 36344 261412 0 0 0 0 1028 1179 0 0 100 0
1 1 696488 40780 36352 261404 0 0 0 44 1012 1193 0 0 100 1
0 0 696488 40780 36352 261404 0 0 0 76 1030 1259 0 0 96 4
0 0 696488 40780 36352 261404 0 0 0 20 1014 1214 0 0 100 0
0 0 696488 40780 36352 261404 0 0 0 0 1177 1340 0 0 100 0
0 0 696488 40780 36352 261404 0 0 0 0 1008 1206 0 0 100 0
[root@localhost ~]# free -m
total used free shared buffers cached
Mem: 1010 970 39 0 35 255
-/+ buffers/cache: 679 330
Swap: 1176 680 496
thanks in adavance!
jimmy
|
|
|
01-22-2008, 02:44 PM
|
#2
|
Senior Member
Registered: May 2006
Distribution: BeOS, BSD, Caldera, CTOS, Debian, LFS, Mac, Mandrake, Red Hat, Slackware, Solaris, SuSE
Posts: 1,761
Rep:
|
What is the output df -h on your drive(s)?
Kind of curious what that zombied process is, er was?
Last edited by bsdunix; 01-22-2008 at 02:45 PM.
|
|
|
01-22-2008, 03:43 PM
|
#3
|
LQ Veteran
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,390
|
Probably tasks waiting for I/O - that's quite a bit of swap in use. Interferes with (normal) I/O when it has to go to disk. Try this to find out how many tasks you have (probably) waiting on I/O
Code:
top -b -n 1 | awk '{if (NR <=7) print; else if ($8 == "D") {print; count++} } END {print "Total status D: "count}'
Last edited by syg00; 01-22-2008 at 03:47 PM.
|
|
|
01-22-2008, 03:53 PM
|
#4
|
Member
Registered: Jun 2006
Posts: 132
Original Poster
Rep:
|
[root@localhost ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 4.9G 3.9G 725M 85% /
none 506M 0 506M 0% /dev/shm
/dev/sdb1 9.7G 6.2G 3.0G 68% /disk01
/dev/sdb3 3.2G 2.9G 191M 94% /disk02
/dev/sda2 2.5G 1.8G 507M 79% /home
/dev/sdb2 3.9G 3.2G 469M 88% /var
192.168.1.102:/opt/backup/jira/backup/110
27G 4.5G 22G 18% /var/backup/jira
[root@localhost ~]# top -b -n 1 | awk '{if (NR <=7) print; else if ($8 == "D") {print; count++} } END {print "Total status D: "count}'
top - 16:52:18 up 399 days, 21:46, 4 users, load average: 7.43, 8.11, 8.36
Tasks: 126 total, 1 running, 124 sleeping, 0 stopped, 1 zombie
Cpu(s): 4.1% us, 1.2% sy, 0.0% ni, 92.8% id, 1.7% wa, 0.0% hi, 0.1% si
Mem: 1034728k total, 955824k used, 78904k free, 20300k buffers
Swap: 1204864k total, 714892k used, 489972k free, 262452k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8424 root 18 0 4080 376 372 D 0.0 0.0 0:00.05 du
9303 root 18 0 3608 376 372 D 0.0 0.0 0:00.04 du
17062 root 18 0 4808 376 372 D 0.0 0.0 0:00.10 du
28209 root 18 0 5388 412 408 D 0.0 0.0 0:00.08 find
11535 nmai 17 0 4920 428 424 D 0.0 0.0 0:00.06 find
20033 root 18 0 5572 452 448 D 0.0 0.0 0:00.05 find
1363 root 18 0 4760 552 552 D 0.0 0.1 0:00.05 find
Total status D: 7
|
|
|
01-22-2008, 04:05 PM
|
#5
|
Member
Registered: Jun 2006
Posts: 132
Original Poster
Rep:
|
[root@localhost ~]# kill -9 1363
[root@localhost ~]# kill -9 1363
[root@localhost ~]# top -b -n 1 | awk '{if (NR <=7) print; else if ($8 == "D") {print; count++} } END {print "Total status D: "count}'
top - 17:04:19 up 399 days, 21:58, 4 users, load average: 7.19, 7.39, 7.79
Tasks: 126 total, 1 running, 124 sleeping, 0 stopped, 1 zombie
Cpu(s): 4.1% us, 1.2% sy, 0.0% ni, 92.8% id, 1.7% wa, 0.0% hi, 0.1% si
Mem: 1034728k total, 971024k used, 63704k free, 25532k buffers
Swap: 1204864k total, 713696k used, 491168k free, 265760k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8424 root 18 0 4080 376 372 D 0.0 0.0 0:00.05 du
9303 root 18 0 3608 376 372 D 0.0 0.0 0:00.04 du
17062 root 18 0 4808 376 372 D 0.0 0.0 0:00.10 du
28209 root 18 0 5388 412 408 D 0.0 0.0 0:00.08 find
11535 nmai 17 0 4920 428 424 D 0.0 0.0 0:00.06 find
20033 root 18 0 5572 452 448 D 0.0 0.0 0:00.05 find
1363 root 18 0 4760 552 552 D 0.0 0.1 0:00.05 find
Total status D: 7
I even can't kill them, what I can do? I don't want to restart server.
|
|
|
01-23-2008, 10:49 AM
|
#6
|
Member
Registered: Mar 2007
Distribution: RHEL, Fedora, Open Suse
Posts: 151
Rep:
|
If you have "sysstat" you can see all this info in almost one go. For more details :
#man sar
|
|
|
01-23-2008, 01:55 PM
|
#7
|
Member
Registered: Jun 2006
Posts: 132
Original Poster
Rep:
|
[root@localhost ~]# sar
Linux 2.6.9-5.0.3.ELsmp (localhost.localdomain) 01/23/2008
12:00:02 AM CPU %user %nice %system %iowait %idle
12:10:02 AM all 3.50 0.01 1.94 1.54 93.01
12:20:02 AM all 3.57 0.00 1.94 1.44 93.05
12:30:01 AM all 3.22 0.00 1.94 1.43 93.41
12:40:01 AM all 3.21 0.00 1.93 1.41 93.45
12:50:01 AM all 3.57 0.00 1.93 1.41 93.09
01:00:01 AM all 3.48 0.01 1.89 1.40 93.22
01:10:01 AM all 3.46 0.00 1.94 1.32 93.27
01:20:01 AM all 3.58 0.00 1.92 1.41 93.08
01:30:01 AM all 3.52 0.01 1.97 1.41 93.10
01:40:01 AM all 3.59 0.01 1.94 1.50 92.96
01:50:01 AM all 3.47 0.00 2.00 1.36 93.17
02:00:01 AM all 3.33 0.00 1.92 1.37 93.38
02:10:01 AM all 3.24 0.01 1.95 1.42 93.38
|
|
|
01-24-2008, 07:04 PM
|
#8
|
LQ Guru
Registered: Aug 2004
Location: Sydney
Distribution: Rocky 9.x
Posts: 18,440
|
As syg00 pointed out, over half your swap is being used.. that's not good. If it's always at that sort of level, adding more RAM is indicated. Functionally, swap is just a temp extension of RAM, it should normally (for decent perf) be mostly unused.
From your posts there, swap usage is slowly increaseing ....
You've got several disks there that are nearly full; you need to purge unwanted stuff and/or backup & remove some stuff before they fill up.
Alternately, add more disks.
|
|
|
02-08-2008, 12:28 AM
|
#9
|
Member
Registered: Sep 2003
Posts: 32
Rep:
|
Quote:
Originally Posted by jimmyjiang
[root@localhost ~]# kill -9 1363
[root@localhost ~]# kill -9 1363
[root@localhost ~]# top -b -n 1 | awk '{if (NR <=7) print; else if ($8 == "D") {print; count++} } END {print "Total status D: "count}'
top - 17:04:19 up 399 days, 21:58, 4 users, load average: 7.19, 7.39, 7.79
Tasks: 126 total, 1 running, 124 sleeping, 0 stopped, 1 zombie
Cpu(s): 4.1% us, 1.2% sy, 0.0% ni, 92.8% id, 1.7% wa, 0.0% hi, 0.1% si
Mem: 1034728k total, 971024k used, 63704k free, 25532k buffers
Swap: 1204864k total, 713696k used, 491168k free, 265760k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8424 root 18 0 4080 376 372 D 0.0 0.0 0:00.05 du
9303 root 18 0 3608 376 372 D 0.0 0.0 0:00.04 du
17062 root 18 0 4808 376 372 D 0.0 0.0 0:00.10 du
28209 root 18 0 5388 412 408 D 0.0 0.0 0:00.08 find
11535 nmai 17 0 4920 428 424 D 0.0 0.0 0:00.06 find
20033 root 18 0 5572 452 448 D 0.0 0.0 0:00.05 find
1363 root 18 0 4760 552 552 D 0.0 0.1 0:00.05 find
Total status D: 7
I even can't kill them, what I can do? I don't want to restart server.
|
Unfortunately if the process is in a "D" dead state, you have to reboot the machine  No kill command will remove it.
|
|
|
All times are GMT -5. The time now is 11:28 PM.
|
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
|
Latest Threads
LQ News
|
|