Hi all,
I just wonder if anyone can give some hint regarding my problem. I have a server based on RHEL 4.4
What I have noticed is that server load average is raising in approx. 7-10 days by approx. 7, even if has very low CPU utilization, memory utilization on approx 80% and low on I/O stats.
Linux apollo 2.6.9-67.ELsmp #1 SMP Wed Nov 7 13:58:04 EST 2007 i686 i686 i386 GNU/Linux
Code:
top - 15:47:06 up 12 days, 2:36, 26 users, load average: 13.03, 12.75, 12.43
Tasks: 375 total, 1 running, 370 sleeping, 4 stopped, 0 zombie
Cpu0 : 5.0% us, 4.3% sy, 0.0% ni, 90.4% id, 0.0% wa, 0.3% hi, 0.0% si
Cpu1 : 5.3% us, 3.3% sy, 0.0% ni, 91.4% id, 0.0% wa, 0.0% hi, 0.0% si
Cpu2 : 1.0% us, 1.3% sy, 0.0% ni, 97.4% id, 0.3% wa, 0.0% hi, 0.0% si
Cpu3 : 2.3% us, 1.0% sy, 0.0% ni, 95.0% id, 1.7% wa, 0.0% hi, 0.0% si
Mem: 4151256k total, 3284184k used, 867072k free, 186272k buffers
Swap: 2031608k total, 0k used, 2031608k free, 2403068k cached
PID USER PR NI %CPU TIME+ %MEM VIRT RES SHR S COMMAND
21366 blasztst 16 0 3 0:00.08 0.1 5808 2964 1608 S bba_shelf_acces
21315 witasmic 17 0 2 0:00.05 0.0 3192 1828 788 S ppt_comptest_wo
21349 witasmic 18 0 2 0:00.05 0.1 5632 3000 1620 S bba_shelf_acces
9768 fijalwit 15 0 1 0:00.33 0.1 16192 5864 3572 S clearmake
21276 root 16 0 1 0:00.08 0.0 2612 1288 868 R top
21313 witasmic 18 0 1 0:00.02 0.1 4220 2456 1568 S ppt_comptest
21355 witasmic 15 0 1 0:00.02 0.1 5864 2080 1712 S ssh
21385 blasztst 20 0 1 0:00.02 0.1 16392 5076 4104 S cleartool
20243 witasmic 15 0 0 0:03.80 0.0 8668 1860 1220 S sshd
21363 root 16 0 0 0:00.01 0.1 8820 2304 1844 S sshd
1 root 16 0 0 0:00.94 0.0 3612 548 468 S init
2 root RT 0 0 0:02.19 0.0 0 0 0 S migration/0
3 root 34 19 0 0:01.20 0.0 0 0 0 S ksoftirqd/0
4 root RT 0 0 0:02.00 0.0 0 0 0 S migration/1
5 root 34 19 0 0:01.50 0.0 0 0 0 S ksoftirqd/1
6 root RT 0 0 0:01.59 0.0 0 0 0 S migration/2
7 root 34 19 0 0:02.78 0.0 0 0 0 S ksoftirqd/2
8 root RT 0 0 0:01.53 0.0 0 0 0 S migration/3
9 root 34 19 0 0:02.83 0.0 0 0 0 S ksoftirqd/3
10 root 5 -10 0 0:00.04 0.0 0 0 0 S events/0
11 root 5 -10 0 0:00.02 0.0 0 0 0 S events/1
12 root 5 -10 0 0:00.00 0.0 0 0 0 S events/2
13 root 5 -10 0 0:00.02 0.0 0 0 0 S events/3
14 root 6 -10 0 0:00.01 0.0 0 0 0 S khelper
15 root 15 -10 0 0:00.00 0.0 0 0 0 S kacpid
35 root 5 -10 0 0:00.00 0.0 0 0 0 S kblockd/0
36 root 5 -10 0 0:00.00 0.0 0 0 0 S kblockd/1
37 root 5 -10 0 0:00.00 0.0 0 0 0 S kblockd/2
38 root 5 -10 0 0:00.00 0.0 0 0 0 S kblockd/3
39 root 15 0 0 0:00.91 0.0 0 0 0 S khubd
56 root 20 0 0 0:00.00 0.0 0 0 0 S pdflush
57 root 15 0 0 0:23.67 0.0 0 0 0 S pdflush
58 root 15 0 0 0:06.52 0.0 0 0 0 S kswapd0
59 root 13 -10 0 0:00.00 0.0 0 0 0 S aio/0
60 root 13 -10 0 0:00.00 0.0 0 0 0 S aio/1
61 root 5 -10 0 0:00.00 0.0 0 0 0 S aio/2
Code:
-bash-3.00# uname -a
Linux apollo 2.6.9-67.ELsmp #1 SMP Wed Nov 7 13:58:04 EST 2007 i686 i686 i386 GNU/Linux
-bash-3.00# w
15:44:41 up 12 days, 2:33, 26 users, load average: 12.74, 12.65, 12.36
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
witasmic pts/0 ww018476 02Nov10 6:06m 1.57s 1.57s bash
soczypat pts/1 ww018314 02Nov10 7days 45.68s 45.59s konsole
witasmic pts/2 ww018476 28Oct10 31:13m 1.21s 1.76s sshd: witasmic [priv]
luznyjak pts/3 wl037368 15:31 0.00s 0.01s 0.01s sshd: luznyjak [priv]
witasmic pts/5 ww018476 Thu12 28:34m 0.10s 0.84s sshd: witasmic [priv]
fijalwit pts/6 ww019158 03Nov10 34.00s 1.87s 0.00s /bin/sh ./adsl_load.sh
witasmic pts/8 ww018476 02Nov10 28:29m 0.22s 1.16s sshd: witasmic [priv]
witasmic pts/10 ww018476 Mon08 5.00s 0.77s 0.22s sshd: witasmic [priv]
soczypat pts/4 - 02Nov10 3:31m 0.17s 0.17s /bin/bash
ostrokrz pts/9 ww018557 15:38 5:44 0.04s 0.04s bash
furkigeo pts/12 ww016441 Thu12 6:17m 0.24s 0.24s -bash
soczypat pts/11 - 02Nov10 4:26m 0.55s 0.35s /bin/bash
wojtcseb pts/7 ww019916 Mon08 27:36m 0.09s 0.08s bash
kaczmpaw pts/17 ww020265 12:29 17:41 0.14s 0.13s bash
sajaapaw pts/18 ww020710 Mon10 32:14 0.62s 0.60s bash
soczypat pts/13 - Fri16 1:02m 0.06s 0.03s ssh root@172.22.48.18
kaczmpaw pts/21 ww020265 11:59 2:08m 0.20s 0.19s ssh root@172.22.48.9
bohdamic pts/22 ww012377 Mon10 3:14m 0.19s 0.18s bash
chamimar pts/24 ww013996 Mon10 22:10 0.26s 0.10s -bash
wojtcseb pts/26 ww019916 Mon12 5:50m 3.48s 3.43s mc -c
wojtcseb pts/23 ww019916 Mon12 4:39m 0.10s 0.09s bash
witasmic pts/20 ww018476 11:20 2:50m 0.09s 0.08s bash
soczypat pts/15 - Mon16 3:31m 0.11s 0.11s /bin/bash
soczypat pts/16 - 08:40 7:02m 0.03s 0.01s ssh root@172.22.48.17
kaczmpaw pts/27 ww020265 11:59 2:41m 0.34s 0.29s ssh root@172.22.48.9
bohdamic pts/29 ww012377 12:19 26:07 0.10s 0.08s bash
Code:
-bash-3.00# vmstat 2 30
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 0 0 775840 181692 2471348 0 0 1 16 4 9 1 0 99 0
1 0 0 763488 181704 2472896 0 0 0 592 1561 944 9 11 78 1
2 0 0 765352 181716 2473664 0 0 0 1190 1562 858 5 6 88 2
0 0 0 780584 181728 2474432 0 0 0 1000 2065 1655 8 7 82 3
2 0 0 769960 181744 2475196 0 0 0 1236 1810 1618 15 11 72 1
2 0 0 779112 181756 2476484 0 0 0 1254 2483 2093 11 11 76 2
0 0 0 778656 181772 2476468 0 0 0 640 3270 3731 8 9 81 2
1 0 0 775328 181780 2478020 0 0 0 1736 1972 2567 9 9 80 2
0 0 0 778016 181784 2478276 0 0 0 674 1682 1598 12 7 80 1
1 1 0 750496 181784 2479836 0 0 0 1262 2294 3454 21 11 69 0
0 0 0 788416 181788 2477752 0 0 0 860 1921 2893 12 3 81 4
0 0 0 786624 181792 2479048 0 0 0 718 1728 2739 8 8 82 2
0 0 0 785856 181800 2479820 0 0 0 1508 1649 943 11 6 81 2
0 0 0 760448 181840 2505000 0 0 0 13280 2021 2544 2 4 81 13
0 0 0 759200 181852 2506548 0 0 10 1318 2241 1789 7 8 83 2
0 1 0 753992 181876 2511724 0 0 4 2734 2547 2626 6 5 86 3
4 0 0 757936 181888 2511452 0 0 14 318 2614 3148 3 5 91 1
2 0 0 757680 181892 2511448 0 0 0 384 1446 1950 2 5 93 1
1 0 0 752944 181896 2511704 0 0 0 152 2341 5851 12 17 71 0
2 0 0 752424 181896 2511964 0 0 2 1078 1858 4724 16 22 61 1
0 0 0 738856 181900 2511180 0 0 0 888 1901 3962 16 11 72 2
0 0 0 736296 181916 2515324 0 0 0 2 1638 1327 5 2 92 0
1 0 0 737192 181920 2515580 0 0 2 216 1733 1575 15 3 82 1
2 0 0 730728 181928 2515832 0 0 0 2850 1373 1530 23 5 69 3
0 0 0 741832 181940 2516600 0 0 0 386 1350 2567 14 8 77 0
0 0 0 742416 181944 2516596 0 0 0 544 1242 2382 6 6 88 1
1 0 0 738192 181944 2516596 0 0 0 0 2299 2914 9 6 85 0
2 0 0 742544 181956 2517104 0 0 0 646 2781 4673 10 7 83 1
2 0 0 736824 181948 2522832 0 0 2 3308 2694 5544 8 9 77 6
0 0 0 746568 181852 2515388 0 0 216 2970 2517 3742 12 5 72 11
Utilization of the CPU almoust never goes more than 10% (zenoos core performance graphs attached)
Server is used as compilator / build machine to build aplication from source code stored in IBM Rational clearcase envinronment, and as a test server to test those builds with network equipment.
Does anyone can tell what could be causing this behaviour?
Big thanks in advance for help