LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 11-09-2010, 09:05 AM   #1
xelor81
LQ Newbie
 
Registered: Sep 2010
Posts: 3

Rep: Reputation: 0
RHEL 4.4 - load average raising every 7-10 days


Hi all,

I just wonder if anyone can give some hint regarding my problem. I have a server based on RHEL 4.4

What I have noticed is that server load average is raising in approx. 7-10 days by approx. 7, even if has very low CPU utilization, memory utilization on approx 80% and low on I/O stats.

Linux apollo 2.6.9-67.ELsmp #1 SMP Wed Nov 7 13:58:04 EST 2007 i686 i686 i386 GNU/Linux

Code:
top - 15:47:06 up 12 days,  2:36, 26 users,  load average: 13.03, 12.75, 12.43
Tasks: 375 total,   1 running, 370 sleeping,   4 stopped,   0 zombie
Cpu0  :  5.0% us,  4.3% sy,  0.0% ni, 90.4% id,  0.0% wa,  0.3% hi,  0.0% si
Cpu1  :  5.3% us,  3.3% sy,  0.0% ni, 91.4% id,  0.0% wa,  0.0% hi,  0.0% si
Cpu2  :  1.0% us,  1.3% sy,  0.0% ni, 97.4% id,  0.3% wa,  0.0% hi,  0.0% si
Cpu3  :  2.3% us,  1.0% sy,  0.0% ni, 95.0% id,  1.7% wa,  0.0% hi,  0.0% si
Mem:   4151256k total,  3284184k used,   867072k free,   186272k buffers
Swap:  2031608k total,        0k used,  2031608k free,  2403068k cached

  PID USER      PR  NI %CPU    TIME+  %MEM  VIRT  RES  SHR S COMMAND
21366 blasztst  16   0    3   0:00.08  0.1  5808 2964 1608 S bba_shelf_acces
21315 witasmic  17   0    2   0:00.05  0.0  3192 1828  788 S ppt_comptest_wo
21349 witasmic  18   0    2   0:00.05  0.1  5632 3000 1620 S bba_shelf_acces
 9768 fijalwit  15   0    1   0:00.33  0.1 16192 5864 3572 S clearmake
21276 root      16   0    1   0:00.08  0.0  2612 1288  868 R top
21313 witasmic  18   0    1   0:00.02  0.1  4220 2456 1568 S ppt_comptest
21355 witasmic  15   0    1   0:00.02  0.1  5864 2080 1712 S ssh
21385 blasztst  20   0    1   0:00.02  0.1 16392 5076 4104 S cleartool
20243 witasmic  15   0    0   0:03.80  0.0  8668 1860 1220 S sshd
21363 root      16   0    0   0:00.01  0.1  8820 2304 1844 S sshd
    1 root      16   0    0   0:00.94  0.0  3612  548  468 S init
    2 root      RT   0    0   0:02.19  0.0     0    0    0 S migration/0
    3 root      34  19    0   0:01.20  0.0     0    0    0 S ksoftirqd/0
    4 root      RT   0    0   0:02.00  0.0     0    0    0 S migration/1
    5 root      34  19    0   0:01.50  0.0     0    0    0 S ksoftirqd/1
    6 root      RT   0    0   0:01.59  0.0     0    0    0 S migration/2
    7 root      34  19    0   0:02.78  0.0     0    0    0 S ksoftirqd/2
    8 root      RT   0    0   0:01.53  0.0     0    0    0 S migration/3
    9 root      34  19    0   0:02.83  0.0     0    0    0 S ksoftirqd/3
   10 root       5 -10    0   0:00.04  0.0     0    0    0 S events/0
   11 root       5 -10    0   0:00.02  0.0     0    0    0 S events/1
   12 root       5 -10    0   0:00.00  0.0     0    0    0 S events/2
   13 root       5 -10    0   0:00.02  0.0     0    0    0 S events/3
   14 root       6 -10    0   0:00.01  0.0     0    0    0 S khelper
   15 root      15 -10    0   0:00.00  0.0     0    0    0 S kacpid
   35 root       5 -10    0   0:00.00  0.0     0    0    0 S kblockd/0
   36 root       5 -10    0   0:00.00  0.0     0    0    0 S kblockd/1
   37 root       5 -10    0   0:00.00  0.0     0    0    0 S kblockd/2
   38 root       5 -10    0   0:00.00  0.0     0    0    0 S kblockd/3
   39 root      15   0    0   0:00.91  0.0     0    0    0 S khubd
   56 root      20   0    0   0:00.00  0.0     0    0    0 S pdflush
   57 root      15   0    0   0:23.67  0.0     0    0    0 S pdflush
   58 root      15   0    0   0:06.52  0.0     0    0    0 S kswapd0
   59 root      13 -10    0   0:00.00  0.0     0    0    0 S aio/0
   60 root      13 -10    0   0:00.00  0.0     0    0    0 S aio/1
   61 root       5 -10    0   0:00.00  0.0     0    0    0 S aio/2
Code:
-bash-3.00# uname -a
Linux apollo 2.6.9-67.ELsmp #1 SMP Wed Nov 7 13:58:04 EST 2007 i686 i686 i386 GNU/Linux
-bash-3.00# w
 15:44:41 up 12 days,  2:33, 26 users,  load average: 12.74, 12.65, 12.36
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
witasmic pts/0    ww018476   02Nov10  6:06m  1.57s  1.57s bash
soczypat pts/1    ww018314   02Nov10  7days 45.68s 45.59s konsole
witasmic pts/2    ww018476   28Oct10 31:13m  1.21s  1.76s sshd: witasmic [priv]
luznyjak pts/3    wl037368   15:31    0.00s  0.01s  0.01s sshd: luznyjak [priv]
witasmic pts/5    ww018476   Thu12   28:34m  0.10s  0.84s sshd: witasmic [priv]
fijalwit pts/6    ww019158   03Nov10 34.00s  1.87s  0.00s /bin/sh ./adsl_load.sh
witasmic pts/8    ww018476   02Nov10 28:29m  0.22s  1.16s sshd: witasmic [priv]
witasmic pts/10   ww018476   Mon08    5.00s  0.77s  0.22s sshd: witasmic [priv]
soczypat pts/4    -                02Nov10  3:31m  0.17s  0.17s /bin/bash
ostrokrz pts/9    ww018557   15:38    5:44   0.04s  0.04s bash
furkigeo pts/12   ww016441   Thu12    6:17m  0.24s  0.24s -bash
soczypat pts/11   -          02Nov10  4:26m  0.55s  0.35s /bin/bash
wojtcseb pts/7    ww019916   Mon08   27:36m  0.09s  0.08s bash
kaczmpaw pts/17   ww020265   12:29   17:41   0.14s  0.13s bash
sajaapaw pts/18   ww020710   Mon10   32:14   0.62s  0.60s bash
soczypat pts/13   -          Fri16    1:02m  0.06s  0.03s ssh root@172.22.48.18
kaczmpaw pts/21   ww020265   11:59    2:08m  0.20s  0.19s ssh root@172.22.48.9
bohdamic pts/22   ww012377   Mon10    3:14m  0.19s  0.18s bash
chamimar pts/24   ww013996   Mon10   22:10   0.26s  0.10s -bash
wojtcseb pts/26   ww019916   Mon12    5:50m  3.48s  3.43s mc -c
wojtcseb pts/23   ww019916   Mon12    4:39m  0.10s  0.09s bash
witasmic pts/20   ww018476   11:20    2:50m  0.09s  0.08s bash
soczypat pts/15   -          Mon16    3:31m  0.11s  0.11s /bin/bash
soczypat pts/16   -          08:40    7:02m  0.03s  0.01s ssh root@172.22.48.17
kaczmpaw pts/27   ww020265   11:59    2:41m  0.34s  0.29s ssh root@172.22.48.9
bohdamic pts/29   ww012377   12:19   26:07   0.10s  0.08s bash
Code:
-bash-3.00# vmstat 2 30
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0      0 775840 181692 2471348    0    0     1    16    4     9  1  0 99  0
 1  0      0 763488 181704 2472896    0    0     0   592 1561   944  9 11 78  1
 2  0      0 765352 181716 2473664    0    0     0  1190 1562   858  5  6 88  2
 0  0      0 780584 181728 2474432    0    0     0  1000 2065  1655  8  7 82  3
 2  0      0 769960 181744 2475196    0    0     0  1236 1810  1618 15 11 72  1
 2  0      0 779112 181756 2476484    0    0     0  1254 2483  2093 11 11 76  2
 0  0      0 778656 181772 2476468    0    0     0   640 3270  3731  8  9 81  2
 1  0      0 775328 181780 2478020    0    0     0  1736 1972  2567  9  9 80  2
 0  0      0 778016 181784 2478276    0    0     0   674 1682  1598 12  7 80  1
 1  1      0 750496 181784 2479836    0    0     0  1262 2294  3454 21 11 69  0
 0  0      0 788416 181788 2477752    0    0     0   860 1921  2893 12  3 81  4
 0  0      0 786624 181792 2479048    0    0     0   718 1728  2739  8  8 82  2
 0  0      0 785856 181800 2479820    0    0     0  1508 1649   943 11  6 81  2
 0  0      0 760448 181840 2505000    0    0     0 13280 2021  2544  2  4 81 13
 0  0      0 759200 181852 2506548    0    0    10  1318 2241  1789  7  8 83  2
 0  1      0 753992 181876 2511724    0    0     4  2734 2547  2626  6  5 86  3
 4  0      0 757936 181888 2511452    0    0    14   318 2614  3148  3  5 91  1
 2  0      0 757680 181892 2511448    0    0     0   384 1446  1950  2  5 93  1
 1  0      0 752944 181896 2511704    0    0     0   152 2341  5851 12 17 71  0
 2  0      0 752424 181896 2511964    0    0     2  1078 1858  4724 16 22 61  1
 0  0      0 738856 181900 2511180    0    0     0   888 1901  3962 16 11 72  2
 0  0      0 736296 181916 2515324    0    0     0     2 1638  1327  5  2 92  0
 1  0      0 737192 181920 2515580    0    0     2   216 1733  1575 15  3 82  1
 2  0      0 730728 181928 2515832    0    0     0  2850 1373  1530 23  5 69  3
 0  0      0 741832 181940 2516600    0    0     0   386 1350  2567 14  8 77  0
 0  0      0 742416 181944 2516596    0    0     0   544 1242  2382  6  6 88  1
 1  0      0 738192 181944 2516596    0    0     0     0 2299  2914  9  6 85  0
 2  0      0 742544 181956 2517104    0    0     0   646 2781  4673 10  7 83  1
 2  0      0 736824 181948 2522832    0    0     2  3308 2694  5544  8  9 77  6
 0  0      0 746568 181852 2515388    0    0   216  2970 2517  3742 12  5 72 11
Utilization of the CPU almoust never goes more than 10% (zenoos core performance graphs attached)

Server is used as compilator / build machine to build aplication from source code stored in IBM Rational clearcase envinronment, and as a test server to test those builds with network equipment.

Does anyone can tell what could be causing this behaviour?
Big thanks in advance for help
 
Old 11-10-2010, 01:08 AM   #2
xelor81
LQ Newbie
 
Registered: Sep 2010
Posts: 3

Original Poster
Rep: Reputation: 0
I forgot to attach zenoss performance screenshot in the previous post. So i attach it here.
Attached Thumbnails
Click image for larger version

Name:	zenoss.JPG
Views:	20
Size:	113.4 KB
ID:	5158  
 
Old 11-10-2010, 03:59 AM   #3
davinders
LQ Newbie
 
Registered: Nov 2009
Posts: 25

Rep: Reputation: 1
Hi...

memory looks ok because most of the memory in cache....please tell what is the load average normally..

Load average any particular day like monday......first day of week...

Which application is ruuning on this server & check his prerequiste..
 
Old 11-17-2010, 05:36 AM   #4
xelor81
LQ Newbie
 
Registered: Sep 2010
Posts: 3

Original Poster
Rep: Reputation: 0
Load Average normaly is about 2. On weekend is lower - same during the night time.

There few aplication running on this server ie.
- samba
- ssh
- sendmail
- clearcase
- vsftpd
- nfs server
- Xorg with Gnome

Server just after reboot with approx 6 user logged in generated load average approx = 1.8

ClearCase prerequisites are:
Code:
Minimum requirements
    * For Rational ClearCase client hosts:
          o 32 MB main memory
          o 300 MB disk space
    * For Rational ClearCase server hosts:
          o 64 MB memory
          o 2 GB disk space
          o 128 processes on hosts running Linux(TM) or the UNIX(R) system
          o 600 file descriptors per host (not per process) running Linux or the UNIX system
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
load average ? mario.almeida Linux - General 5 08-03-2009 08:39 AM
apache consuming cpu/raising server load elnacho Linux - Server 3 04-10-2009 04:29 AM
Please Help me in my load average black-code Linux - Server 3 04-08-2009 01:17 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 10:33 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration