LinuxQuestions.org - interactive shell very slow

- Linux - Software (https://www.linuxquestions.org/questions/linux-software-2/)

- - interactive shell very slow (https://www.linuxquestions.org/questions/linux-software-2/interactive-shell-very-slow-602653/)

interactive shell very slow

I have a problem with our servers every once in awhile. They are red hat 4. This particular case, the server has 2x dual core Opteron processors and 26 gb of RAM and I've dug around, without finding a cause. Very odd.

The ssh shell performance is horrible.

Just doing an ls of one file is very slow. About 1.5 seconds when it should be about .002 seconds.

The disk IO is non existent according to sar and top.

Anyone have ideas?

uname:

Code:

2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:56:28 EST 2006 x86_64 x86_64 x86_64 GNU/Linux

Top shows:

Code:

top - 23:19:51 up 3 days, 10:57,  2 users,  load average: 1.61, 1.60, 1.69

Tasks: 127 total,  2 running, 125 sleeping,  0 stopped,  0 zombie

Cpu0  :  0.0% us,  7.0% sy,  0.0% ni, 93.0% id,  0.0% wa,  0.0% hi,  0.0% si

Cpu1  :  0.2% us, 47.7% sy,  0.0% ni, 52.1% id,  0.0% wa,  0.0% hi,  0.0% si

Cpu2  :  0.0% us,  0.0% sy,  0.0% ni, 100.0% id,  0.0% wa,  0.0% hi,  0.0% si

Cpu3  :  0.0% us,  0.0% sy, 100.0% ni,  0.0% id,  0.0% wa,  0.0% hi,  0.0% si

Mem:  24751368k total, 24703604k used,    47764k free,  702860k buffers

Swap: 32536648k total,        0k used, 32536648k free, 17603284k cached



  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                              

 9496 bhsu      30  5 5119m 4.8g  38m R 99.9 20.5  4337:51 mantle                                

26179 root      21  0  8444 1140  852 R 48.9  0.0  0:31.48 top                                    

 7705 root      17  0  8036 1200  712 S  6.5  0.0  57:29.45 pim                                    

 7704 root      0 -20 14040 2296 1524 S  5.8  0.0 401:41.92 lim                                    

    1 root      16  0  4752  572  476 S  0.0  0.0  0:00.81 init

Did you try telnet? Or ftp? Something unencrypted and character-based.
For ftp, check if ls is also slow and then check a transfer.
Could be network related, or host related (encryption layer, general problem..)

Same problems on the actual console as an ssh shell.

Notice top is at 48% which isn't right.

It's not using any swap.. very unusual. Normally Linux always has a small amt of RAM free (as here) and a small amt of swap used. Zero swap used is practically unheard of.

Quote:

Originally Posted by chrism01 (Post 2972685)

It's not using any swap.. very unusual. Normally Linux always has a small amt of RAM free (as here) and a small amt of swap used. Zero swap used is practically unheard of.

Here's what free gives me on my machine:

Quote:

total used free shared buffers cached
Mem: 508360 429964 78396 0 19484 224988
-/+ buffers/cache: 185492 322868
Swap: 538168 0 538168

Depending on swapiness, Linux will try the ram used for buffers first.

infinity005, have you physical access or only network access?

edit:
oops I just see this:

9496 bhsu 30 5 5119m 4.8g 38m R 99.9 20.5 4337:51 mantle

That's a lot of memory!!

Mem: 24751368k total, 24703604k used, 47764k free, 702860k buffers

eeek! 24Gb! That's also some memory!! Should be fine without swap.

Can you stop this process for a moment?

The process completed and the system responds normally again.

We have another similar machine that was running mantle with no performance issues.

...why when this system runs mantle and the interactive performance as well as processes using a strange amount of CPU is beyond me at this point.

Is there any additional debugging that I can enable somewhere?

The thing is 24G is lots of RAM, but 47M free is tiny by comparison and zero swap is not right. Even when a Unix/Linux system is just ticking over it always uses a small amt of swap, prob just for bookkeeping purposes (ie so it knows where swap has got to).
As soon as a prog on that box tries to use some more RAM, it's burn that 47M real fast and slow down. Unix/Linux needs some free RAM at all times to move processes/data around eg into/out of the cpu (run list)

Quote:

Originally Posted by chrism01 (Post 2980308)

I would probably agree if I couldn't find a lot of evidence that shows otherwise. Granted this example doesn't show 0, but close. The next one is a new server and shows 0. I'm not sure why swap used is 0 and I'm not sure how to debug why.

Another server:

Code:

top - 22:22:44 up 52 days,  7:51, 10 users,  load average: 1.94, 1.88, 1.81

Tasks: 142 total,  2 running, 138 sleeping,  0 stopped,  2 zombie

Cpu0  :  2.5%us,  2.5%sy, 57.8%ni, 35.7%id,  0.8%wa,  0.0%hi,  0.7%si,  0.0%st

Cpu1  :  4.8%us, 11.4%sy, 17.6%ni, 53.0%id, 11.6%wa,  0.1%hi,  1.5%si,  0.0%st

Mem:  6120104k total,  6050876k used,    69228k free,  252464k buffers

Swap: 12582900k total,      44k used, 12582856k free,  3573568k cached



  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  P COMMAND                              

23440 user1    37  12  8424 5964  508 R  97  0.1  54250:45 1 john                                

 3888 mysql    15  0  765m 675m 6612 S    6 11.3  3864:17 1 mysqld                              

 3140 user1    15  0 10724 1264  892 R    2  0.0  0:00.01 0 top      

 6083 root      5 -10  302m 222m 204m S    2  3.7  6403:37 1 vmware-vmx

A new server:

Code:

top - 16:21:52 up  4:20,  5 users,  load average: 0.00, 0.00, 0.00

Tasks:  73 total,  1 running,  71 sleeping,  0 stopped,  1 zombie

Cpu(s):  0.0%us,  0.3%sy,  0.0%ni, 99.1%id,  0.3%wa,  0.1%hi,  0.3%si,  0.0%st

Mem:  6180488k total,  302040k used,  5878448k free,    57848k buffers

Swap: 12582904k total,        0k used, 12582904k free,  152060k cached



  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                

    1 root      15  0  6120  688  568 S    0  0.0  0:01.42 init                                  

    2 root      RT  0    0    0    0 S    0  0.0  0:00.00 migration/0                            

    3 root      34  19    0    0    0 S    0  0.0  0:00.00 ksoftirqd/0                            

    4 root      RT  0    0    0    0 S    0  0.0  0:00.00 watchdog/0

Mem: 24751368k total, 24703604k used, 47764k free, 702860k buffers

You have 47M free but 702M buffered. So you really have 749M free from what I see.
Depending on swapiness, it will prefer to use the buffer rather than swap on disk.
When buffers drop under a certain level as free then it should use the swap.

Quote:

Originally Posted by chrism01 (Post 2972685)

It's not using any swap.. very unusual. Normally Linux always has a small amt of RAM free (as here) and a small amt of swap used. Zero swap used is practically unheard of.

Quote:

Originally Posted by chrism01 (Post 2980308)

I'm amazed people seem to think that zero swap is unheard of... Just logged into my 24/7 (for the most part) bittorrent (40 torrents seeding currently, DL at 80KB/s and upload at 20KB/s) and media streaming server located at home:

Code:

server@talkie:~$ free

            total      used      free    shared    buffers    cached

Mem:        92180      89080      3100          0      2584      48408

-/+ buffers/cache:      38088      54092

Swap:      597232          0    597232



server@talkie:~$ uptime

 07:00:09 up 6 days, 17:36,  1 user,  load average: 0.36, 0.24, 0.19

Most 'used' RAM is just cache that can be overwritten easily if needed. There is very little RAM that is required if you are managing your server correctly. 24GB of RAM should be able to handle the requirements of Linux pretty well, since my 96MB RAM seems to run everything dandily :)

Well, that's very interesting. I've never seen a server that had absolutely zero swap used and zero load avgs. As I say, normally just basic background activity generates some numbers....
You learn something everyday...

If anyone is interested in how load avg is calc and what it really means, here is a good 2 part article: http://www.teamquest.com/resources/g...ay/5/index.htm