Hi all,

I wasn't sure where exactly to post this because it's not super clear what the underlying issue is. Apologies if this is in the wrong place.

I recently purchased an old server box to use as a home server. It's currently connected to my home network over WiFi (temporary). It uses an Intel Xeon 5670 processor and runs Manjaro.

when I ssh into this machine, there is a very long delay between when I type and when it shows up on my screen (~1-2 seconds).

My first suspicion was the delay time caused by the WiFi card. However, when I directly ping my server, the typical delay is ~4ms. Also when I download content to my server from the web, I get speeds on the order of 20MB/s.

I next suspected that the bottleneck might be the encryption cipher underlying the SSH packets. However, when I run gotop over SSH, all 12 CPU cores show between 0% and 1% usage. Furthermore, when I SSH with different encryption ciphers, I see no noticeable improvement.

What's even stranger is that when I host a server off this box using nginx, it serves HTML/JS in <50ms. So it seems to be limited to SSH. This really makes me think that encryption is the bottleneck. But I can't see why that would be the case. The X5670 is certainly an older processor, but not so old that encrypting packets should freeze up the system.

The SSH delay makes doing work on the box basically impossible. Any hints would be greatly appreciated.

Update: the server downloaded a 100MB file over HTTPS in 20 seconds. So encryption by itself does not seem to be a bottleneck...

Chances are very good it's a DNS issue related to a reverse-lookup. Open your /etc/ssh/sshd_config file, and look for the UseDNS directive, and set the value to no. Then either reboot or restart the ssh daemon ("sudo service sshd restart").

1 members found this post helpful.

Ah, forgot to mention that I already tried that as recommended by some other threads

Ok....so now check/toggle the "UsePam" and "UseLogin" directives. And you're sure you rebooted or restarted the ssh daemon after making the DNS change?

1 members found this post helpful.

Am I understanding you correct in that every keystroke in an SSH session takes 1-2 seconds to display onscreen? Or does this happen only during the logon phase?

2 members found this post helpful.

I tried this as well and it didn't help. I both restarted the sshd service (Manjaro uses systemd) and rebooted and nothing helps

Yes. It's every keystroke

I have done some further testing which has only made me more confused. I tried tested upload/download speeds over HTTPS from my server box to an external server and get like 20Mb/s DL 10Mb/s UL. I tested upload/download speeds over HTTPS on my laptop and get comparable results.

But then when I transfer 100 Mb of random information from my server box to my laptop using `scp` over my local network using the internal IP address, the transfer rate starts at 3Mb/s before stalling down to 100 kb/s. This makes no sense right? I am not sure how to even begin diagnosing such a strange problem

Trying scp was a good call. It should definitely not perform so poorly, but even 100 kb/s is much better than what you're experiencing at the command line, by at least two orders of magnitude.

It sounds to me like you're seeing excessive packet loss and packets are being retransmitted over and over again. It's the kind of thing you'd expect to see:

• when a host has been misconfigured with two gateways, or
• a host is using the wrong netmask, causing traffic in one direction to be mistakenly sent via a router until an ARP broadcast from the recipient temporarily corrects the path, or
• there's a duplicate IP (or even MAC) address somewhere on the local LAN, or
• when a defective NIC/cable/switch port is involved, or
• if the transmitting host has been mistakenly configured to use jumbo frames, or
• when traffic is going across a VLAN trunk, but one NIC is incapable of handling frames between 1515 or 1518 bytes

But none of that seems to apply, and besides it seems SSH is the only service affected.

Just to clarify: The server is on WiFi, and WiFi only? No dual-homing? And it's on the same IP subnet as the host running the SSH client? Are they both getting IP addresses via DHCP?

2 members found this post helpful.

It's a standard home WiFi router-modem setup. No dual-homing. All devices on my network use the same subnet and use DHCP.

I downloaded iperf3 onto the server and client. It's showing a throughput of ~20 Mb/s. After 24 Mb were transferred, there were only 30 re-transmitted packets, which seems low to me. Though I'm certainly not an expert.

I'm at a total loss here. I'm thinking about maybe reinstalling to see if that fixes anything.

You could try running tcpdump in a console session on the server while connecting via SSH from the other PC:That should give you an idea as to what's going on.

1 members found this post helpful.

Try adding -vvv to your ssh login/connection. From man ssh:

1 members found this post helpful.

What kind of router?? Is it possible that QOS is enabled and is throttling that port? Quick way to try would be to turn off your web server, and configure SSH to use port 443, and try to connect then, and see if the problem follows.

1 members found this post helpful.

That was a great suggestion, I haven't used tcpdump in quite a while.


When I typed the single character "h", it triggered the following entries:

Clearly there are ~1 second delays at two steps in the TCP handshake.

Excuse me while I dust off my old networking textbook to decipher exactly what is happening here

This gives me the following output (I don't know enough about SSH to see a problem here):

Also, the same packets are being acknowledged multiple times. Something, somewhere is causing packet loss.

(As for the SSH log, it seems perfectly normal. This really seems to be a transport or network issue.)

I'd have tcpdump capture an entire SSH session and save it to a file. Capture files can be opened and viewed in WireShark, which makes it much easier to see the various flags, follow the TCP stream etc.

Try:With tcpdump running, connect via SSH, log in and then just exit. The capture file should be informative.

BTW, when you ran iperf, did you run it in TCP mode? If so, it really seems only SSH is affected by this issue.

There are very few network components or mechanisms that can selectively affect TCP traffic on certain ports only. Off the top of my head, I can think of:

• Firewalls
• Policy routing

...and, well, that's about it.

You said you tried dowloading a file via HTTPS from an external host to your server, and that worked just fine. Is it possible that the client (host or software) could be causing this issue? Have you tried logging in via SSH from other devices?

1 members found this post helpful.