LinuxQuestions.org
Did you know LQ has a Linux Hardware Compatibility List?
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices

Reply
 
Search this Thread
Old 11-27-2007, 01:25 AM   #1
infinity005
LQ Newbie
 
Registered: Nov 2007
Posts: 5

Rep: Reputation: 0
interactive shell very slow


I have a problem with our servers every once in awhile. They are red hat 4. This particular case, the server has 2x dual core Opteron processors and 26 gb of RAM and I've dug around, without finding a cause. Very odd.

The ssh shell performance is horrible.

Just doing an ls of one file is very slow. About 1.5 seconds when it should be about .002 seconds.

The disk IO is non existent according to sar and top.

Anyone have ideas?

uname:
Code:
2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:56:28 EST 2006 x86_64 x86_64 x86_64 GNU/Linux
Top shows:

Code:
top - 23:19:51 up 3 days, 10:57,  2 users,  load average: 1.61, 1.60, 1.69
Tasks: 127 total,   2 running, 125 sleeping,   0 stopped,   0 zombie
Cpu0  :  0.0% us,  7.0% sy,  0.0% ni, 93.0% id,  0.0% wa,  0.0% hi,  0.0% si
Cpu1  :  0.2% us, 47.7% sy,  0.0% ni, 52.1% id,  0.0% wa,  0.0% hi,  0.0% si
Cpu2  :  0.0% us,  0.0% sy,  0.0% ni, 100.0% id,  0.0% wa,  0.0% hi,  0.0% si
Cpu3  :  0.0% us,  0.0% sy, 100.0% ni,  0.0% id,  0.0% wa,  0.0% hi,  0.0% si
Mem:  24751368k total, 24703604k used,    47764k free,   702860k buffers
Swap: 32536648k total,        0k used, 32536648k free, 17603284k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                               
 9496 bhsu      30   5 5119m 4.8g  38m R 99.9 20.5   4337:51 mantle                                 
26179 root      21   0  8444 1140  852 R 48.9  0.0   0:31.48 top                                    
 7705 root      17   0  8036 1200  712 S  6.5  0.0  57:29.45 pim                                    
 7704 root       0 -20 14040 2296 1524 S  5.8  0.0 401:41.92 lim                                    
    1 root      16   0  4752  572  476 S  0.0  0.0   0:00.81 init
 
Old 11-27-2007, 06:55 AM   #2
nx5000
Senior Member
 
Registered: Sep 2005
Location: Out
Posts: 3,307

Rep: Reputation: 52
Did you try telnet? Or ftp? Something unencrypted and character-based.
For ftp, check if ls is also slow and then check a transfer.
Could be network related, or host related (encryption layer, general problem..)
 
Old 11-27-2007, 02:33 PM   #3
infinity005
LQ Newbie
 
Registered: Nov 2007
Posts: 5

Original Poster
Rep: Reputation: 0
Same problems on the actual console as an ssh shell.

Notice top is at 48% which isn't right.
 
Old 11-27-2007, 05:59 PM   #4
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,232

Rep: Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024
It's not using any swap.. very unusual. Normally Linux always has a small amt of RAM free (as here) and a small amt of swap used. Zero swap used is practically unheard of.
 
Old 11-28-2007, 05:52 AM   #5
nx5000
Senior Member
 
Registered: Sep 2005
Location: Out
Posts: 3,307

Rep: Reputation: 52
Quote:
Originally Posted by chrism01 View Post
It's not using any swap.. very unusual. Normally Linux always has a small amt of RAM free (as here) and a small amt of swap used. Zero swap used is practically unheard of.
Here's what free gives me on my machine:

Quote:
total used free shared buffers cached
Mem: 508360 429964 78396 0 19484 224988
-/+ buffers/cache: 185492 322868
Swap: 538168 0 538168
Depending on swapiness, Linux will try the ram used for buffers first.


infinity005, have you physical access or only network access?


edit:
oops I just see this:

9496 bhsu 30 5 5119m 4.8g 38m R 99.9 20.5 4337:51 mantle

That's a lot of memory!!

Mem: 24751368k total, 24703604k used, 47764k free, 702860k buffers

eeek! 24Gb! That's also some memory!! Should be fine without swap.


Can you stop this process for a moment?
 
Old 11-28-2007, 01:04 PM   #6
infinity005
LQ Newbie
 
Registered: Nov 2007
Posts: 5

Original Poster
Rep: Reputation: 0
The process completed and the system responds normally again.

We have another similar machine that was running mantle with no performance issues.

...why when this system runs mantle and the interactive performance as well as processes using a strange amount of CPU is beyond me at this point.
 
Old 12-04-2007, 11:33 AM   #7
infinity005
LQ Newbie
 
Registered: Nov 2007
Posts: 5

Original Poster
Rep: Reputation: 0
Is there any additional debugging that I can enable somewhere?
 
Old 12-04-2007, 05:44 PM   #8
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,232

Rep: Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024
The thing is 24G is lots of RAM, but 47M free is tiny by comparison and zero swap is not right. Even when a Unix/Linux system is just ticking over it always uses a small amt of swap, prob just for bookkeeping purposes (ie so it knows where swap has got to).
As soon as a prog on that box tries to use some more RAM, it's burn that 47M real fast and slow down. Unix/Linux needs some free RAM at all times to move processes/data around eg into/out of the cpu (run list)
 
Old 12-05-2007, 12:27 AM   #9
infinity005
LQ Newbie
 
Registered: Nov 2007
Posts: 5

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by chrism01 View Post
The thing is 24G is lots of RAM, but 47M free is tiny by comparison and zero swap is not right. Even when a Unix/Linux system is just ticking over it always uses a small amt of swap, prob just for bookkeeping purposes (ie so it knows where swap has got to).
As soon as a prog on that box tries to use some more RAM, it's burn that 47M real fast and slow down. Unix/Linux needs some free RAM at all times to move processes/data around eg into/out of the cpu (run list)
I would probably agree if I couldn't find a lot of evidence that shows otherwise. Granted this example doesn't show 0, but close. The next one is a new server and shows 0. I'm not sure why swap used is 0 and I'm not sure how to debug why.

Another server:
Code:
top - 22:22:44 up 52 days,  7:51, 10 users,  load average: 1.94, 1.88, 1.81
Tasks: 142 total,   2 running, 138 sleeping,   0 stopped,   2 zombie
Cpu0  :  2.5%us,  2.5%sy, 57.8%ni, 35.7%id,  0.8%wa,  0.0%hi,  0.7%si,  0.0%st
Cpu1  :  4.8%us, 11.4%sy, 17.6%ni, 53.0%id, 11.6%wa,  0.1%hi,  1.5%si,  0.0%st
Mem:   6120104k total,  6050876k used,    69228k free,   252464k buffers
Swap: 12582900k total,       44k used, 12582856k free,  3573568k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  P COMMAND                              
23440 user1     37  12  8424 5964  508 R   97  0.1  54250:45 1 john                                 
 3888 mysql     15   0  765m 675m 6612 S    6 11.3   3864:17 1 mysqld                               
 3140 user1     15   0 10724 1264  892 R    2  0.0   0:00.01 0 top       
 6083 root       5 -10  302m 222m 204m S    2  3.7   6403:37 1 vmware-vmx
A new server:
Code:
top - 16:21:52 up  4:20,  5 users,  load average: 0.00, 0.00, 0.00
Tasks:  73 total,   1 running,  71 sleeping,   0 stopped,   1 zombie
Cpu(s):  0.0%us,  0.3%sy,  0.0%ni, 99.1%id,  0.3%wa,  0.1%hi,  0.3%si,  0.0%st
Mem:   6180488k total,   302040k used,  5878448k free,    57848k buffers
Swap: 12582904k total,        0k used, 12582904k free,   152060k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                
    1 root      15   0  6120  688  568 S    0  0.0   0:01.42 init                                   
    2 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/0                            
    3 root      34  19     0    0    0 S    0  0.0   0:00.00 ksoftirqd/0                            
    4 root      RT   0     0    0    0 S    0  0.0   0:00.00 watchdog/0
 
Old 12-05-2007, 05:53 AM   #10
nx5000
Senior Member
 
Registered: Sep 2005
Location: Out
Posts: 3,307

Rep: Reputation: 52
Mem: 24751368k total, 24703604k used, 47764k free, 702860k buffers

You have 47M free but 702M buffered. So you really have 749M free from what I see.
Depending on swapiness, it will prefer to use the buffer rather than swap on disk.
When buffers drop under a certain level as free then it should use the swap.
 
Old 12-05-2007, 01:02 PM   #11
Aeiri
Member
 
Registered: Feb 2004
Posts: 307

Rep: Reputation: 30
Quote:
Originally Posted by chrism01 View Post
It's not using any swap.. very unusual. Normally Linux always has a small amt of RAM free (as here) and a small amt of swap used. Zero swap used is practically unheard of.
Quote:
Originally Posted by chrism01 View Post
The thing is 24G is lots of RAM, but 47M free is tiny by comparison and zero swap is not right. Even when a Unix/Linux system is just ticking over it always uses a small amt of swap, prob just for bookkeeping purposes (ie so it knows where swap has got to).
As soon as a prog on that box tries to use some more RAM, it's burn that 47M real fast and slow down. Unix/Linux needs some free RAM at all times to move processes/data around eg into/out of the cpu (run list)
I'm amazed people seem to think that zero swap is unheard of... Just logged into my 24/7 (for the most part) bittorrent (40 torrents seeding currently, DL at 80KB/s and upload at 20KB/s) and media streaming server located at home:

Code:
server@talkie:~$ free
             total       used       free     shared    buffers     cached
Mem:         92180      89080       3100          0       2584      48408
-/+ buffers/cache:      38088      54092
Swap:       597232          0     597232

server@talkie:~$ uptime
 07:00:09 up 6 days, 17:36,  1 user,  load average: 0.36, 0.24, 0.19
Most 'used' RAM is just cache that can be overwritten easily if needed. There is very little RAM that is required if you are managing your server correctly. 24GB of RAM should be able to handle the requirements of Linux pretty well, since my 96MB RAM seems to run everything dandily
 
Old 12-05-2007, 06:13 PM   #12
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,232

Rep: Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024
Well, that's very interesting. I've never seen a server that had absolutely zero swap used and zero load avgs. As I say, normally just basic background activity generates some numbers....
You learn something everyday...

If anyone is interested in how load avg is calc and what it really means, here is a good 2 part article: http://www.teamquest.com/resources/g...ay/5/index.htm
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
shell script in non interactive way googlix Linux - General 6 11-13-2007 09:18 AM
how to know if the shell is interactive or not? anupamsr Linux - Newbie 2 01-20-2007 07:31 AM
interactive and non-interactive shell linuxjamil Programming 3 09-03-2006 08:42 PM
Run interactive program in shell escargot22026 Programming 4 08-14-2004 11:59 PM
[Shell scripting] interactive and crontab pb maxance Programming 4 06-10-2004 12:00 PM


All times are GMT -5. The time now is 06:23 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration