Places > Connect To Server issues

MHewitt · 07-09-2014, 09:48 AM

Hi All.

First, here's my setup: three servers all running Deb 7.5, (Named tst1, tst2, tst3) and one laptop running Deb 7.5 (Named max).

The sshd_config and ssh_config files are the same on all three servers as well as the laptop.

Authorized Keys and HostKeys are copied to each of the servers and the laptop (I can ssh fine through a terminal back and forth between all boxes).

I can add a place via Places > Connect To Server (using ssh port 22) from the laptop to both tst2 and tst3, but I cannot add the place to tst1. I receive a Host Key Verification Failed error. (Keep in mind that I CAN ssh via terminal to the same box.)

So what could be causing this error?

I've tried removing all host and auth keys from both tst1 and max and adding them again to no avail. I'm sure someone out there smarter than I has run into this and found a solution, right???

schneidz · 07-09-2014, 09:53 AM

try using sshfs from the command line and see if there are the same problems.

maybe run the server as sshd -d -d -d to see if it is trying to do something weird ?

MHewitt · 07-09-2014, 10:26 AM

Hey schneidz. The servers are actually in production so I'm hesitant to add any new packages to them (sshfs). But I did try restarting the server using sshd -d -d -d. I saw two debug errors that indicate Incorrect RSA1 identifier. Here's the debug output:

debug2: load_server_config: filename /etc/ssh/sshd_config
debug2: load_server_config: done config len = 775
debug2: parse_server_config: config /etc/ssh/sshd_config len 775
debug3: /etc/ssh/sshd_config:5 setting Port 22
debug3: /etc/ssh/sshd_config:9 setting Protocol 2
debug3: /etc/ssh/sshd_config:11 setting HostKey /etc/ssh/ssh_host_rsa_key
debug3: /etc/ssh/sshd_config:12 setting HostKey /etc/ssh/ssh_host_dsa_key
debug3: /etc/ssh/sshd_config:13 setting HostKey /etc/ssh/ssh_host_ecdsa_key
debug3: /etc/ssh/sshd_config:15 setting UsePrivilegeSeparation yes
debug3: /etc/ssh/sshd_config:18 setting KeyRegenerationInterval 3600
debug3: /etc/ssh/sshd_config:19 setting ServerKeyBits 768
debug3: /etc/ssh/sshd_config:22 setting SyslogFacility AUTH
debug3: /etc/ssh/sshd_config:23 setting LogLevel INFO
debug3: /etc/ssh/sshd_config:26 setting LoginGraceTime 120
debug3: /etc/ssh/sshd_config:27 setting PermitRootLogin yes
debug3: /etc/ssh/sshd_config:28 setting StrictModes no
debug3: /etc/ssh/sshd_config:30 setting RSAAuthentication yes
debug3: /etc/ssh/sshd_config:31 setting PubkeyAuthentication yes
debug3: /etc/ssh/sshd_config:32 setting AuthorizedKeysFile %h/.ssh/authorized_keys
debug3: /etc/ssh/sshd_config:35 setting IgnoreRhosts yes
debug3: /etc/ssh/sshd_config:37 setting RhostsRSAAuthentication no
debug3: /etc/ssh/sshd_config:39 setting HostbasedAuthentication no
debug3: /etc/ssh/sshd_config:44 setting PermitEmptyPasswords no
debug3: /etc/ssh/sshd_config:48 setting ChallengeResponseAuthentication no
debug3: /etc/ssh/sshd_config:51 setting PasswordAuthentication yes
debug3: /etc/ssh/sshd_config:63 setting X11Forwarding yes
debug3: /etc/ssh/sshd_config:64 setting X11DisplayOffset 10
debug3: /etc/ssh/sshd_config:65 setting PrintMotd no
debug3: /etc/ssh/sshd_config:66 setting PrintLastLog yes
debug3: /etc/ssh/sshd_config:67 setting TCPKeepAlive yes
debug3: /etc/ssh/sshd_config:71 setting Banner /etc/ssh/ssh.banner
debug3: /etc/ssh/sshd_config:74 setting AcceptEnv LANG LC_*
debug3: /etc/ssh/sshd_config:76 setting Subsystem sftp /usr/lib/openssh/sftp-server
debug3: /etc/ssh/sshd_config:87 setting UsePAM no
debug1: sshd version OpenSSH_6.0p1 Debian-4+deb7u1
debug3: Incorrect RSA1 identifier
debug1: read PEM private key done: type RSA
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
debug1: private host key: #0 type 1 RSA
debug3: Incorrect RSA1 identifier
debug1: read PEM private key done: type DSA
debug1: Checking blacklist file /usr/share/ssh/blacklist.DSA-1024
debug1: Checking blacklist file /etc/ssh/blacklist.DSA-1024
debug1: private host key: #1 type 2 DSA
debug3: Incorrect RSA1 identifier
debug1: read PEM private key done: type ECDSA
debug1: Checking blacklist file /usr/share/ssh/blacklist.ECDSA-256
debug1: Checking blacklist file /etc/ssh/blacklist.ECDSA-256
debug1: private host key: #2 type 3 ECDSA
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-d'
debug1: rexec_argv[2]='-d'
debug1: rexec_argv[3]='-d'
debug3: oom_adjust_setup
Set /proc/self/oom_score_adj from 0 to -1000
debug2: fd 3 setting O_NONBLOCK
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug2: fd 4 setting O_NONBLOCK
debug3: sock_set_v6only: set socket 4 IPV6_V6ONLY
debug1: Bind to port 22 on ::.
Server listening on :: port 22.

When I attempted to connect, I saw this output on the server:

debug3: fd 5 is not O_NONBLOCK
debug1: Server will not fork when running in debugging mode.
debug3: send_rexec_state: entering fd = 8 config len 775
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: inetd sockets after dupping: 3, 3
Connection from 192.168.1.210 port 50419
debug1: Client protocol version 2.0; client software version OpenSSH_6.0p1 Debian-4+deb7u1
debug1: match: OpenSSH_6.0p1 Debian-4+deb7u1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.0p1 Debian-4+deb7u1
debug2: fd 3 setting O_NONBLOCK
debug2: Network child is on pid 17135
debug3: preauth child monitor started
debug3: privsep user:group 113:65534 [preauth]
debug1: permanently_set_uid: 113/65534 [preauth]
debug1: list_hostkey_types: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 [preauth]
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256 [preauth]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se [preauth]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se [preauth]
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 [preauth]
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 [preauth]
debug2: kex_parse_kexinit: none,zlib@openssh.com [preauth]
debug2: kex_parse_kexinit: none,zlib@openssh.com [preauth]
debug2: kex_parse_kexinit: [preauth]
debug2: kex_parse_kexinit: [preauth]
debug2: kex_parse_kexinit: first_kex_follows 0 [preauth]
debug2: kex_parse_kexinit: reserved 0 [preauth]
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 [preauth]
debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01@openssh...00@openssh.com,ssh-dss [preauth]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se [preauth]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se [preauth]
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 [preauth]
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 [preauth]
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib [preauth]
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib [preauth]
debug2: kex_parse_kexinit: [preauth]
debug2: kex_parse_kexinit: [preauth]
debug2: kex_parse_kexinit: first_kex_follows 0 [preauth]
debug2: kex_parse_kexinit: reserved 0 [preauth]
debug2: mac_setup: found hmac-md5 [preauth]
debug1: kex: client->server aes128-ctr hmac-md5 none [preauth]
debug2: mac_setup: found hmac-md5 [preauth]
debug1: kex: server->client aes128-ctr hmac-md5 none [preauth]
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
debug3: mm_key_sign entering [preauth]
debug3: mm_request_send entering: type 5 [preauth]
debug3: mm_key_sign: waiting for MONITOR_ANS_SIGN [preauth]
debug3: mm_request_receive_expect entering: type 6 [preauth]
debug3: mm_request_receive entering [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 5
debug3: mm_answer_sign
debug3: mm_answer_sign: signature 0x7f43764d8610(99)
debug3: mm_request_send entering: type 6
debug2: monitor_read: 5 used once, disabling now
debug2: kex_derive_keys [preauth]
debug2: set_newkeys: mode 1 [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
Connection closed by 192.168.1.210 [preauth]
debug1: do_cleanup [preauth]
debug3: mm_request_receive entering
debug1: do_cleanup
debug1: Killing privsep child 17135

Not sure what I'm looking for by the way of errors.

schneidz · 07-09-2014, 10:34 AM

i'm not sure either. just for shiggles, maybe we can get the same sshd -d -d -d output from a working server for comparison ?

MHewitt · 07-09-2014, 10:38 AM

Here's one from a working server:

starting with sshd -d -d -d

debug2: load_server_config: filename /etc/ssh/sshd_config
debug2: load_server_config: done config len = 775
debug2: parse_server_config: config /etc/ssh/sshd_config len 775
debug3: /etc/ssh/sshd_config:5 setting Port 22
debug3: /etc/ssh/sshd_config:9 setting Protocol 2
debug3: /etc/ssh/sshd_config:11 setting HostKey /etc/ssh/ssh_host_rsa_key
debug3: /etc/ssh/sshd_config:12 setting HostKey /etc/ssh/ssh_host_dsa_key
debug3: /etc/ssh/sshd_config:13 setting HostKey /etc/ssh/ssh_host_ecdsa_key
debug3: /etc/ssh/sshd_config:15 setting UsePrivilegeSeparation yes
debug3: /etc/ssh/sshd_config:18 setting KeyRegenerationInterval 3600
debug3: /etc/ssh/sshd_config:19 setting ServerKeyBits 768
debug3: /etc/ssh/sshd_config:22 setting SyslogFacility AUTH
debug3: /etc/ssh/sshd_config:23 setting LogLevel INFO
debug3: /etc/ssh/sshd_config:26 setting LoginGraceTime 120
debug3: /etc/ssh/sshd_config:27 setting PermitRootLogin yes
debug3: /etc/ssh/sshd_config:28 setting StrictModes no
debug3: /etc/ssh/sshd_config:30 setting RSAAuthentication yes
debug3: /etc/ssh/sshd_config:31 setting PubkeyAuthentication yes
debug3: /etc/ssh/sshd_config:32 setting AuthorizedKeysFile %h/.ssh/authorized_keys
debug3: /etc/ssh/sshd_config:35 setting IgnoreRhosts yes
debug3: /etc/ssh/sshd_config:37 setting RhostsRSAAuthentication no
debug3: /etc/ssh/sshd_config:39 setting HostbasedAuthentication no
debug3: /etc/ssh/sshd_config:44 setting PermitEmptyPasswords no
debug3: /etc/ssh/sshd_config:48 setting ChallengeResponseAuthentication no
debug3: /etc/ssh/sshd_config:51 setting PasswordAuthentication yes
debug3: /etc/ssh/sshd_config:63 setting X11Forwarding yes
debug3: /etc/ssh/sshd_config:64 setting X11DisplayOffset 10
debug3: /etc/ssh/sshd_config:65 setting PrintMotd no
debug3: /etc/ssh/sshd_config:66 setting PrintLastLog yes
debug3: /etc/ssh/sshd_config:67 setting TCPKeepAlive yes
debug3: /etc/ssh/sshd_config:71 setting Banner /etc/ssh/ssh.banner
debug3: /etc/ssh/sshd_config:74 setting AcceptEnv LANG LC_*
debug3: /etc/ssh/sshd_config:76 setting Subsystem sftp /usr/lib/openssh/sftp-server
debug3: /etc/ssh/sshd_config:87 setting UsePAM no
debug1: sshd version OpenSSH_6.0p1 Debian-4+deb7u1
debug3: Incorrect RSA1 identifier
debug1: read PEM private key done: type RSA
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
debug1: private host key: #0 type 1 RSA
debug3: Incorrect RSA1 identifier
debug1: read PEM private key done: type DSA
debug1: Checking blacklist file /usr/share/ssh/blacklist.DSA-1024
debug1: Checking blacklist file /etc/ssh/blacklist.DSA-1024
debug1: private host key: #1 type 2 DSA
debug3: Incorrect RSA1 identifier
debug1: read PEM private key done: type ECDSA
debug1: Checking blacklist file /usr/share/ssh/blacklist.ECDSA-256
debug1: Checking blacklist file /etc/ssh/blacklist.ECDSA-256
debug1: private host key: #2 type 3 ECDSA
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-d'
debug1: rexec_argv[2]='-d'
debug1: rexec_argv[3]='-d'
debug3: oom_adjust_setup
Set /proc/self/oom_score_adj from 0 to -1000
debug2: fd 3 setting O_NONBLOCK
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug2: fd 4 setting O_NONBLOCK
debug3: sock_set_v6only: set socket 4 IPV6_V6ONLY
debug1: Bind to port 22 on ::.
Server listening on :: port 22.

And the connection debug on the server I get NO debug output.

schneidz · 07-09-2014, 10:46 AM

fyi sshfs is a client program.

the only diff between the working and non-working debug output seems to be the additional output when you attempt to login.

hopefully someone more knowledgeable than me will come by with better ideas.

while you were debugging were you using ssh or connecting via nautilus ?

instead of copying keys maybe you should re-run ssh-keygen ?

MHewitt · 07-09-2014, 10:59 AM

I was using ssh (Presumably that is what Places > Connect To Server uses when you select ssh.) I don't have nautilus installed on the client nor the server.

I actually tried regenerating host keys earlier, but tried it again just to verify. Removed old keys, removed openssh-server, reinstalled openssh-server, regenerated keys.

Nothing. This is strange to me because presumably all three servers are identical..apparently not.

schneidz · 07-09-2014, 11:26 AM

by nautilus i assumed thats what the 'places->connect to server' was based on. try debugging using the gui menu as well as command-line ssh and see if there are any obvious output that could be ruled out.

MHewitt · 07-11-2014, 04:31 PM

OK. For the purposes of documentation, I thought I'd post the solution to this problem that I finally stumbled up. Probably a newbie mistake, but nonetheless here it is:

I actually discovered through trial and error that in fact when connecting through Places > Connect To Server, you are actually using the host key information stored in your /user/.ssh/known_hosts file, even if you are using the User Name : root and Password : XXXX in the Connect To Server box.

Therefore, clear out your known_hosts file in your /home/USERNAME/.ssh/known_hosts file and you should then be good to go.

In fact, I cleared the known_hosts file in all machines for both the root and USERNAME accounts just to be sure.

Problem solved!