interactive shell very slow

infinity005 · 11-27-2007, 01:25 AM

I have a problem with our servers every once in awhile. They are red hat 4. This particular case, the server has 2x dual core Opteron processors and 26 gb of RAM and I've dug around, without finding a cause. Very odd.

The ssh shell performance is horrible.

Just doing an ls of one file is very slow. About 1.5 seconds when it should be about .002 seconds.

The disk IO is non existent according to sar and top.

Anyone have ideas?

uname:

Code:

2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:56:28 EST 2006 x86_64 x86_64 x86_64 GNU/Linux

Top shows:

Code:

top - 23:19:51 up 3 days, 10:57,  2 users,  load average: 1.61, 1.60, 1.69
Tasks: 127 total,   2 running, 125 sleeping,   0 stopped,   0 zombie
Cpu0  :  0.0% us,  7.0% sy,  0.0% ni, 93.0% id,  0.0% wa,  0.0% hi,  0.0% si
Cpu1  :  0.2% us, 47.7% sy,  0.0% ni, 52.1% id,  0.0% wa,  0.0% hi,  0.0% si
Cpu2  :  0.0% us,  0.0% sy,  0.0% ni, 100.0% id,  0.0% wa,  0.0% hi,  0.0% si
Cpu3  :  0.0% us,  0.0% sy, 100.0% ni,  0.0% id,  0.0% wa,  0.0% hi,  0.0% si
Mem:  24751368k total, 24703604k used,    47764k free,   702860k buffers
Swap: 32536648k total,        0k used, 32536648k free, 17603284k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                               
 9496 bhsu      30   5 5119m 4.8g  38m R 99.9 20.5   4337:51 mantle                                 
26179 root      21   0  8444 1140  852 R 48.9  0.0   0:31.48 top                                    
 7705 root      17   0  8036 1200  712 S  6.5  0.0  57:29.45 pim                                    
 7704 root       0 -20 14040 2296 1524 S  5.8  0.0 401:41.92 lim                                    
    1 root      16   0  4752  572  476 S  0.0  0.0   0:00.81 init

nx5000 · 11-27-2007, 06:55 AM

Did you try telnet? Or ftp? Something unencrypted and character-based.
For ftp, check if ls is also slow and then check a transfer.
Could be network related, or host related (encryption layer, general problem..)

infinity005 · 11-27-2007, 02:33 PM

Same problems on the actual console as an ssh shell.

Notice top is at 48% which isn't right.

chrism01 · 11-27-2007, 05:59 PM

It's not using any swap.. very unusual. Normally Linux always has a small amt of RAM free (as here) and a small amt of swap used. Zero swap used is practically unheard of.

nx5000 · 11-28-2007, 05:52 AM

Quote:

Originally Posted by chrism01

It's not using any swap.. very unusual. Normally Linux always has a small amt of RAM free (as here) and a small amt of swap used. Zero swap used is practically unheard of.

Here's what free gives me on my machine:

Quote:

total used free shared buffers cached
Mem: 508360 429964 78396 0 19484 224988
-/+ buffers/cache: 185492 322868
Swap: 538168 0 538168

Depending on swapiness, Linux will try the ram used for buffers first.

infinity005, have you physical access or only network access?

edit:
oops I just see this:

9496 bhsu 30 5 5119m 4.8g 38m R 99.9 20.5 4337:51 mantle

That's a lot of memory!!

Mem: 24751368k total, 24703604k used, 47764k free, 702860k buffers

eeek! 24Gb! That's also some memory!! Should be fine without swap.

Can you stop this process for a moment?

infinity005 · 11-28-2007, 01:04 PM

The process completed and the system responds normally again.

We have another similar machine that was running mantle with no performance issues.

...why when this system runs mantle and the interactive performance as well as processes using a strange amount of CPU is beyond me at this point.

infinity005 · 12-04-2007, 11:33 AM

Is there any additional debugging that I can enable somewhere?

chrism01 · 12-04-2007, 05:44 PM

The thing is 24G is lots of RAM, but 47M free is tiny by comparison and zero swap is not right. Even when a Unix/Linux system is just ticking over it always uses a small amt of swap, prob just for bookkeeping purposes (ie so it knows where swap has got to).
As soon as a prog on that box tries to use some more RAM, it's burn that 47M real fast and slow down. Unix/Linux needs some free RAM at all times to move processes/data around eg into/out of the cpu (run list)

infinity005 · 12-05-2007, 12:27 AM

Quote:

Originally Posted by chrism01

The thing is 24G is lots of RAM, but 47M free is tiny by comparison and zero swap is not right. Even when a Unix/Linux system is just ticking over it always uses a small amt of swap, prob just for bookkeeping purposes (ie so it knows where swap has got to).
As soon as a prog on that box tries to use some more RAM, it's burn that 47M real fast and slow down. Unix/Linux needs some free RAM at all times to move processes/data around eg into/out of the cpu (run list)

I would probably agree if I couldn't find a lot of evidence that shows otherwise. Granted this example doesn't show 0, but close. The next one is a new server and shows 0. I'm not sure why swap used is 0 and I'm not sure how to debug why.

Another server:

Code:

top - 22:22:44 up 52 days,  7:51, 10 users,  load average: 1.94, 1.88, 1.81
Tasks: 142 total,   2 running, 138 sleeping,   0 stopped,   2 zombie
Cpu0  :  2.5%us,  2.5%sy, 57.8%ni, 35.7%id,  0.8%wa,  0.0%hi,  0.7%si,  0.0%st
Cpu1  :  4.8%us, 11.4%sy, 17.6%ni, 53.0%id, 11.6%wa,  0.1%hi,  1.5%si,  0.0%st
Mem:   6120104k total,  6050876k used,    69228k free,   252464k buffers
Swap: 12582900k total,       44k used, 12582856k free,  3573568k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  P COMMAND                              
23440 user1     37  12  8424 5964  508 R   97  0.1  54250:45 1 john                                 
 3888 mysql     15   0  765m 675m 6612 S    6 11.3   3864:17 1 mysqld                               
 3140 user1     15   0 10724 1264  892 R    2  0.0   0:00.01 0 top       
 6083 root       5 -10  302m 222m 204m S    2  3.7   6403:37 1 vmware-vmx

A new server:

Code:

top - 16:21:52 up  4:20,  5 users,  load average: 0.00, 0.00, 0.00
Tasks:  73 total,   1 running,  71 sleeping,   0 stopped,   1 zombie
Cpu(s):  0.0%us,  0.3%sy,  0.0%ni, 99.1%id,  0.3%wa,  0.1%hi,  0.3%si,  0.0%st
Mem:   6180488k total,   302040k used,  5878448k free,    57848k buffers
Swap: 12582904k total,        0k used, 12582904k free,   152060k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                
    1 root      15   0  6120  688  568 S    0  0.0   0:01.42 init                                   
    2 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/0                            
    3 root      34  19     0    0    0 S    0  0.0   0:00.00 ksoftirqd/0                            
    4 root      RT   0     0    0    0 S    0  0.0   0:00.00 watchdog/0

nx5000 · 12-05-2007, 05:53 AM

Mem: 24751368k total, 24703604k used, 47764k free, 702860k buffers

You have 47M free but 702M buffered. So you really have 749M free from what I see.
Depending on swapiness, it will prefer to use the buffer rather than swap on disk.
When buffers drop under a certain level as free then it should use the swap.

Aeiri · 12-05-2007, 01:02 PM

Quote:

Originally Posted by chrism01

It's not using any swap.. very unusual. Normally Linux always has a small amt of RAM free (as here) and a small amt of swap used. Zero swap used is practically unheard of.

Quote:

Originally Posted by chrism01

The thing is 24G is lots of RAM, but 47M free is tiny by comparison and zero swap is not right. Even when a Unix/Linux system is just ticking over it always uses a small amt of swap, prob just for bookkeeping purposes (ie so it knows where swap has got to).
As soon as a prog on that box tries to use some more RAM, it's burn that 47M real fast and slow down. Unix/Linux needs some free RAM at all times to move processes/data around eg into/out of the cpu (run list)

I'm amazed people seem to think that zero swap is unheard of... Just logged into my 24/7 (for the most part) bittorrent (40 torrents seeding currently, DL at 80KB/s and upload at 20KB/s) and media streaming server located at home:

Code:

server@talkie:~$ free
             total       used       free     shared    buffers     cached
Mem:         92180      89080       3100          0       2584      48408
-/+ buffers/cache:      38088      54092
Swap:       597232          0     597232

server@talkie:~$ uptime
 07:00:09 up 6 days, 17:36,  1 user,  load average: 0.36, 0.24, 0.19

Most 'used' RAM is just cache that can be overwritten easily if needed. There is very little RAM that is required if you are managing your server correctly. 24GB of RAM should be able to handle the requirements of Linux pretty well, since my 96MB RAM seems to run everything dandily

chrism01 · 12-05-2007, 06:13 PM

Well, that's very interesting. I've never seen a server that had absolutely zero swap used and zero load avgs. As I say, normally just basic background activity generates some numbers....
You learn something everyday...

If anyone is interested in how load avg is calc and what it really means, here is a good 2 part article: http://www.teamquest.com/resources/g...ay/5/index.htm