I set up a Ubuntu on a Virtualbox VM instance, using bridged network. External SSH to it is very slow. I am not sure if it's a Ubuntu problem or a Virtualbox problem. I labeled some delay gaps in a debug ssh session:

ssh -v mybox.mydomain.xxx -l myid

OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to mybox.mydomain.xxx [129.186.142.168] port 22.
debug1: Connection established.
debug1: identity file /home/hu/.ssh/identity type -1
debug1: identity file /home/hu/.ssh/id_rsa type 1
debug1: identity file /home/hu/.ssh/id_dsa type -1
debug1: loaded 3 keys
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6p1 Ubuntu-2ubuntu1
debug1: match: OpenSSH_6.6p1 Ubuntu-2ubuntu1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.3
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host 'mybox.mydomain.xxx' is known and matches the RSA host key.
debug1: Found key in /home/eded/.ssh/known_hosts:23
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
(-----pause 12 seconds------)
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Trying private key: /home/hu/.ssh/identity
debug1: Offering public key: /home/hu/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 277
debug1: read PEM private key done: type RSA
(-----pause 5 seconds------)
debug1: Authentication succeeded (publickey).
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env LANG = en_US
(-----pause 31 seconds------)
Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.13.0-24-generic i686)

Any suggestions? Thanks in advance.

joe

These days .xxx is gtld domain name. Might it be that ssh is contacting internet for resolving purposes.

What operating sustem is the host machine running and on which hardware? Is it possible that the CPU is simply over worked?
I've run all kinds of servers virtualised in Virtual Box and not encountered any problems so I don"t think it's an issue with VB itself.

Try adding more verbosity to the output, maybe -vvv. The only thing I've run into in the past that delayed SSH connections (other than slow network connection, overloaded CPU, etc.) is GSSAPIAuthentication, and with enough verbosity in the output it was pretty clear that was the hangup. Adding the following to .ssh/config on the client system cleared it up:

I'll second that - one thing that can slow down validation is DNS resolution for external certificate validation. Uncached DNS lookups can take 10 seconds... You might check the /etc/nsswitch configuration. Perhaps local hosts are getting a DNS lookup before the file checks.

From your log, I would think that maybe the server is doing DNS before local file lookups.

Thank you all for taking your time to reply. Good to learn 'more verbose' trick. Please see below a "more verbose" ssh connection (sorry it is very long, so I attach it in the end).

I didn't have it and added a ".ssh/config" file on my ssh client account that contains:

Host *
GSSAPIAuthentication no

I also edited the ssh server "/etc/ssh/sshd_config", commented out the line to make:

GSSAPIAuthentication no

Followed by a "service ssh reload" command. These all made no difference.

I also looked at my "/etc/nsswitch.conf" file (attached below) but to me "dns" comes after local "file" (or I am wrong understanding the syntax?).

More advice please?

joe

------------------------------------------------------
# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.

passwd: compat
group: compat
shadow: compat

hosts: files dns
networks: files

protocols: db files
services: db files
ethers: db files
rpc: db files

netgroup: nis

------------------------------------------------------
> ssh vvv xxx.xxx.xxx.xxx
OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to xxxx.xxxxxxx.xxx[xxx.xxx.xxx.xxx] port 22.
debug1: Connection established.
debug1: identity file /home/xxxx/.ssh/identity type -1
debug3: Not a RSA1 key file /home/xxxx/.ssh/id_rsa.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /home/xxxx/.ssh/id_rsa type 1
debug1: identity file /home/xxxx/.ssh/id_dsa type -1
debug1: loaded 3 keys

(----------- paused 11 seconds -------------)

debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6p1 Ubuntu-2ubuntu1
debug1: match: OpenSSH_6.6p1 Ubuntu-2ubuntu1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.3
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-e...60@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-e...60@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_init: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_init: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug2: dh_gen_key: priv key bits set: 114/256
debug2: bits set: 505/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug3: check_host_in_hostfile: filename /home/xxxx/.ssh/known_hosts2
debug3: check_host_in_hostfile: filename /etc/ssh/ssh_known_hosts2
debug3: check_host_in_hostfile: filename /home/xxxx/.ssh/known_hosts2
debug3: check_host_in_hostfile: filename /etc/ssh/ssh_known_hosts2
debug3: check_host_in_hostfile: filename /home/xxxx/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 23
debug3: check_host_in_hostfile: filename /home/xxxx/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 23
debug1: Host 'xxxx.xxxxxxxx.xxx' is known and matches the RSA host key.
debug1: Found key in /home/xxxx/.ssh/known_hosts:23
debug2: bits set: 536/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/xxxx/.ssh/identity ((nil))
debug2: key: /home/xxxx/.ssh/id_rsa (0x2b5e2f2c3ab0)
debug2: key: /home/xxxx/.ssh/id_dsa ((nil))

(----------- paused 20 seconds -------------)

debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/xxxx/.ssh/identity
debug3: no such identity: /home/xxxx/.ssh/identity
debug1: Offering public key: /home/xxxx/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 277
debug2: input_userauth_pk_ok: SHA1 fp cb:01:45:28:1c:2c:87:8c:16:1e:41:6b:55:14:12:48:bd:32:4d:6c
debug3: sign_and_send_pubkey
debug1: read PEM private key done: type RSA

(----------- paused 26 seconds -------------)

debug1: Authentication succeeded (publickey).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Entering interactive session.
debug2: callback start
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 0
debug3: tty_make_modes: ospeed 38400
debug3: tty_make_modes: ispeed 38400
debug3: tty_make_modes: 1 3
debug3: tty_make_modes: 2 28
debug3: tty_make_modes: 3 127
debug3: tty_make_modes: 4 21
debug3: tty_make_modes: 5 4
debug3: tty_make_modes: 6 0
debug3: tty_make_modes: 7 0
debug3: tty_make_modes: 8 17
debug3: tty_make_modes: 9 19
debug3: tty_make_modes: 10 26
debug3: tty_make_modes: 12 18
debug3: tty_make_modes: 13 23
debug3: tty_make_modes: 14 22
debug3: tty_make_modes: 18 15
debug3: tty_make_modes: 30 0
debug3: tty_make_modes: 31 0
debug3: tty_make_modes: 32 0
debug3: tty_make_modes: 33 0
debug3: tty_make_modes: 34 0
debug3: tty_make_modes: 35 0
debug3: tty_make_modes: 36 1
debug3: tty_make_modes: 37 0
debug3: tty_make_modes: 38 1
debug3: tty_make_modes: 39 0
debug3: tty_make_modes: 40 0
debug3: tty_make_modes: 41 0
debug3: tty_make_modes: 50 1
debug3: tty_make_modes: 51 1
debug3: tty_make_modes: 52 0
debug3: tty_make_modes: 53 1
debug3: tty_make_modes: 54 1
debug3: tty_make_modes: 55 1
debug3: tty_make_modes: 56 0
debug3: tty_make_modes: 57 0
debug3: tty_make_modes: 58 0
debug3: tty_make_modes: 59 1
debug3: tty_make_modes: 60 1
debug3: tty_make_modes: 61 1
debug3: tty_make_modes: 62 0
debug3: tty_make_modes: 70 1
debug3: tty_make_modes: 71 0
debug3: tty_make_modes: 72 1
debug3: tty_make_modes: 73 0
debug3: tty_make_modes: 74 0
debug3: tty_make_modes: 75 0
debug3: tty_make_modes: 90 1
debug3: tty_make_modes: 91 1
debug3: tty_make_modes: 92 0
debug3: tty_make_modes: 93 0
debug1: Sending environment.
debug3: Ignored env USER
debug3: Ignored env LOGNAME
debug3: Ignored env HOME
debug3: Ignored env PATH
debug3: Ignored env MAIL
debug3: Ignored env SHELL
debug3: Ignored env SSH_CLIENT
debug3: Ignored env SSH_CONNECTION
debug3: Ignored env SSH_TTY
debug3: Ignored env TERM
debug3: Ignored env HOSTTYPE
debug3: Ignored env VENDOR
debug3: Ignored env OSTYPE
debug3: Ignored env MACHTYPE
debug3: Ignored env SHLVL
debug3: Ignored env PWD
debug3: Ignored env GROUP
debug3: Ignored env HOST
debug3: Ignored env REMOTEHOST
debug3: Ignored env LS_COLORS
debug3: Ignored env CVS_RSH
debug3: Ignored env G_BROKEN_FILENAMES
debug3: Ignored env SSH_ASKPASS
debug1: Sending env LANG = en_US
debug2: channel 0: request env confirm 0
debug3: Ignored env LESSOPEN
debug3: Ignored env NXDIR
debug3: Ignored env PAGER
debug3: Ignored env JAVA_HOME
debug3: Ignored env JRE_HOME
debug3: Ignored env HOSTNAME
debug3: Ignored env INPUTRC
debug3: Ignored env PERL5LIB
debug3: Ignored env SVN_EDITOR
debug2: channel 0: request shell confirm 0
debug2: fd 3 setting TCP_NODELAY
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel 0: rcvd adjust 2097152

(----------- paused 42 seconds -------------)

Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.13.0-24-generic i686)
----------
/* end */

This is an update - I seem to have it solved:

First I edited "/etc/nsswitch.conf" to make:

#hosts: files dns
hosts: files

This reduced waiting time from 40-50 seconds to 5 seconds.

Next I edited "/etc/pam.d/sshd" to comment out these 3 lines:

#session optional pam_motd.so motd=/run/motd.dynamic noupdate
#session optional pam_motd.so # [1]
#session optional pam_mail.so standard noenv # [1]

which made me to ssh connect in no time! :-)

Did I do something out of line? Any advice would be appreciated!

joe

Nothing wrong. It is a bit odd that pam_mail.so takes a long time...

The pam_motd.so shouldn't take any time - either the file /etc/motd exists or not, and if it exists it should be printed.

One other thing - you might check your DNS configuration. It really shouldn't take quite so long.

Indeed, now I spend some time to edit the "/etc/network/interfaces" and found there is a wrong IP among other good ones on the dns-nameserver line. However I learned I needed this command to update the settings besides restarting the "init.d/networking":

sudo ifdown eth0 && sudo ifup eth0

Now I have "/etc/nsswitch.conf":

hosts: files dns

and remote ssh to it is as fast as it should be

Thank you all for your time and helps. I used to use redhat; It's only recently I get to ubuntu.

joe