ssh client works on all but one machine

stevena · 05-30-2010, 03:56 AM

Hi, i have RHEL 4 U7 installed on 4 machines, with approximately same configuration. yesterday i was trying to set up public key authentication for ssh on them to be able to run some automated jobs but i had a problem with one particular machine: when used as an ssh client to access the other 3, it kept prompting for a password. the configuration worked on the other 3, they can be accessed from each other with no problem. they can also access machine 1 - the one that has the ssh client problem - too with public key authentication.
when i run ssh -vv on machine one i get the following output :

Code:

[orausr@atsrvnode1 .ssh]$ ssh -vv -o GSSAPIAuthentication=no asyusr@10.16.10.228
OpenSSH_3.9p1, OpenSSL 0.9.7a Feb 19 2003
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 10.16.10.228 [10.16.10.228] port 22.
debug1: Connection established.
debug1: identity file /home/orausr/.ssh/identity type -1
debug1: identity file /home/orausr/.ssh/id_rsa type -1
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug2: key_type_from_name: unknown key type '-----END'
debug1: identity file /home/orausr/.ssh/id_dsa type 2
debug1: Remote protocol version 1.99, remote software version OpenSSH_3.9p1
debug1: match: OpenSSH_3.9p1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_3.9p1
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_init: found hmac-md5
debug1: kex: server->client aes128-cbc hmac-md5 none
debug2: mac_init: found hmac-md5
debug1: kex: client->server aes128-cbc hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug2: dh_gen_key: priv key bits set: 122/256
debug2: bits set: 508/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host '10.16.10.228' is known and matches the RSA host key.
debug1: Found key in /home/orausr/.ssh/known_hosts:7
debug2: bits set: 489/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/orausr/.ssh/identity ((nil))
debug2: key: /home/orausr/.ssh/id_rsa ((nil))
debug2: key: /home/orausr/.ssh/id_dsa (0x552abffba0)
debug1: Authentications that can continue: publickey,gssapi-with-mic,password
debug1: Next authentication method: publickey
debug1: Trying private key: /home/orausr/.ssh/identity
debug1: Trying private key: /home/orausr/.ssh/id_rsa
debug1: Offering public key: /home/orausr/.ssh/id_dsa
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-with-mic,password
debug2: we did not send a packet, disable method
debug1: Next authentication method: password

this is my machine 1's ssh_config file:

Code:

#       $OpenBSD: ssh_config,v 1.19 2003/08/13 08:46:31 markus Exp $

# This is the ssh client system-wide configuration file.  See
# ssh_config(5) for more information.  This file provides defaults for
# users, and the values can be changed in per-user configuration files
# or on the command line.

# Configuration data is parsed as follows:
#  1. command line options
#  2. user-specific file
#  3. system-wide file
# Any configuration value is only changed the first time it is set.
# Thus, host-specific definitions should be at the beginning of the
# configuration file, and defaults at the end.

# Site-wide defaults for various options

# Host *
#   ForwardAgent no
#   ForwardX11 no
#   RhostsRSAAuthentication no
#   RSAAuthentication yes
#   PasswordAuthentication yes
#   HostbasedAuthentication no
#   BatchMode no
#   CheckHostIP yes
#   AddressFamily any
#   ConnectTimeout 0
#   StrictHostKeyChecking ask
#   IdentityFile ~/.ssh/identity
#   IdentityFile ~/.ssh/id_rsa
#   IdentityFile ~/.ssh/id_dsa
#   Port 22
#   Protocol 2,1
#   Cipher 3des
#   Ciphers aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes2
56-cbc
#   EscapeChar ~
Host *
        GSSAPIAuthentication yes
# If this option is set to yes then the remote X11 clients will have full access
# to the local X11 display. As virtually no X11 client supports the untrusted
# mode correctly we set this to yes.
       ForwardX11Trusted yes

please help me figuring out what's wrong.
thank you

druuna · 05-30-2010, 05:12 AM

Hi,

You posted the ssh_client content, I do believe the problem might be in one of the settings in the sshd_config file and not the ssh_config file.

First option that comes to mind: PubkeyAuthentication (should be yes) and AuthorizedKeysFile (probably .ssh/authorized_keys.).

Another thing that comes to mind: Is the created key (the one created on machine 1 and added to the authorized_keys files on the other machines) correct?

Hope this helps.

djsmiley2k · 05-30-2010, 05:16 AM

I had a problem like this once. On one of the machines I had put .ssh/authorised_keys instead of .ssh/authorized_keys.

Worth checking, just in case

Robhogg · 05-30-2010, 05:20 AM

It is also worth checking the permissions on the authorized_keys file - I believe that authentication will fail if "group" or "other" have write permissions to this file.

druuna · 05-30-2010, 05:23 AM

@djsmiley2k and Robhogg:

I doubt if that is the problem, the other 3 machines can connect to each other which shows that the spelling and permissions of files are correct on those 3.

Hope this clears things up

stevena · 05-30-2010, 06:03 AM

Quote:

Originally Posted by druuna

Hi,

You posted the ssh_client content, I do believe the problem might be in one of the settings in the sshd_config file and not the ssh_config file.

First option that comes to mind: PubkeyAuthentication (should be yes) and AuthorizedKeysFile (probably .ssh/authorized_keys.).

thank you for the input.
i haven't touched sshd_config, i assumed public key authentication would be enabled by default. as for AuthorizedKeysFile, isn't it safer to leave it to ~/.ssh/authorized_keys as opposed to a file in a path relative .ssh directory?
anyway for what it's worth, i mangled with the sshd_config on one of the 3 other machines where it works and i tried to connect again with no luck but as you said in your latter post it's irrelevant since these machines can be successfully accessed from each other.
unless you meant sshd_config of machine 1, highly unlikely that it's related but if so please elaborate.

Quote:

Originally Posted by druuna

Another thing that comes to mind: Is the created key (the one created on machine 1 and added to the authorized_keys files on the other machines) correct?
Hope this helps.

well i am deleting the old one, recreating it and copying it on the spot, what could go wrong? what do you mean by correct, this is what i tried

Code:

ssh-keygen -t dsa

and

Code:

ssh-keygen -t rsa

thanks

stevena · 05-30-2010, 06:08 AM

Quote:

Originally Posted by Robhogg

It is also worth checking the permissions on the authorized_keys file - I believe that authentication will fail if "group" or "other" have write permissions to this file.

on one of the 3 other machines

Code:

ls -ld .ssh
drwx------  2 oracle dba 4096 May 30 14:28 .

ls -l .ssh
-rw-------  1 oracle dba  604 May 30 14:28 authorized_keys
-rw-r--r--  1 oracle dba 1332 May 30 14:15 known_hosts

on machine 1

Code:

ls -l .ssh
total 16
-rw-------  1 orausr dba  600 May 30 15:22 authorized_keys
-rw-------  1 orausr dba  672 May 30 19:54 id_dsa
-rw-r--r--  1 orausr dba  604 May 30 19:54 id_dsa.pub
-rw-r--r--  1 orausr dba 1550 May 30 14:50 known_hosts

i suppose these are the right permissions.although the ssh client debugging messages are not much of a help, i never saw a message complaining about incorrect permissions on the remote server being accessed.

druuna · 05-30-2010, 06:41 AM

Hi,

Quote:

i assumed public key authentication would be enabled by default.

Did you check it to be sure?

Quote:

as for AuthorizedKeysFile, isn't it safer to leave it to ~/.ssh/authorized_keys as opposed to a file in a path relative .ssh directory?

.ssh/authorized_keys is indeed the default and I would suggest not changing it.

Did you compare all 4 /etc/ssh/sshd_config files, they should basically be the same. Depending on how you set things up the ListenAddress and/or AllowUsers could be different, the rest is the same.

Quote:

well i am deleting the old one, recreating it and copying it on the spot, what could go wrong?

The file could become corrupt for whatever reason, but if you created a new one already, this probably isn't the case.

I notice you created a rsa and a dsa key. To simplify things, use one of them, not both (and never use rsa1).

Hope this helps.

BTW: The permissions in post #7 are correct.

stevena · 05-31-2010, 12:07 AM

Quote:

Originally Posted by druuna

Hi,

Did you check it to be sure?

.ssh/authorized_keys is indeed the default and I would suggest not changing it.

Did you compare all 4 /etc/ssh/sshd_config files, they should basically be the same. Depending on how you set things up the ListenAddress and/or AllowUsers could be different, the rest is the same.

Hi, this is irrelevant. if the other servers are accessible from each other that would mean that the sshd_config files are ok. why do i need to check them? anyways i did and in fact the only thing that i found to be different was that on one of the servers, the property PubkeyAuthentication was explicitly set to yes( the default is yes too). but that's nothing, the other servers are accessible without this property explicitly set.

Quote:

Originally Posted by druuna

The file could become corrupt for whatever reason, but if you created a new one already, this probably isn't the case.

I notice you created a rsa and a dsa key. To simplify things, use one of them, not both (and never use rsa1).

Hope this helps.

BTW: The permissions in post #7 are correct.

i did not use both dsa and rsa at the same time, only when i got desperate trying to make dsa work, i switched to rsa but faced the same problem.
thank you so much for your input, if you got other ideas i'll be more than happy to check and try mine are exhausted.

druuna · 05-31-2010, 03:39 AM

Hi again,

All the checks I talked about are meant for machine 1. The other 3, which seem to work, could be taken as a reference.

The example output you posted shows that the public key is offered by machine 1 but isn't accepted by the server (either of the 3 other machines):

Quote:

debug1: Authentications that can continue: publickey,gssapi-with-mic,password
debug1: Next authentication method: publickey
debug1: Trying private key: /home/orausr/.ssh/identity
debug1: Trying private key: /home/orausr/.ssh/id_rsa
debug1: Offering public key: /home/orausr/.ssh/id_dsa
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-with-mic,password

After the bold line there should be an accpet from the server, something like this:

Quote:

debug1: Server accepts key: pkalg ssh-rsa blen 408
debug2: input_userauth_pk_ok: fp 23:d3:b3:3c:90:5...............
........

Could it be that your OpenSSH is too old: OpenSSH_3.9p1, OpenSSL 0.9.7a Feb 19 2003? What version are the other 3 machines using?

Another thing that just came to mind: Do you see anything in the logfiles about this (check both machine 1 and the other machine you try to connect to). There could be indicators why this doesn't work.

Not much to go on, but maybe it helps.

stevena · 05-31-2010, 06:14 AM

sshd_config files were different between machine 1 and other machines, so i changed that of machine one to have identical files but still no luck. and on the sshd server log running with full debug mode, there's nothing as well..
i even reinstalled the machine 1 server but still same problem !
openssh version is the same on all machines, nevertheless i will upgrade it and see what happens

druuna · 05-31-2010, 07:03 AM

Hi,

Quote:

sshd_config files were different between machine 1 and other machines, so i changed that of machine one to have identical files but still no luck.

You don't mention it, but you did restart the ssh daemon after changing the sshd_config file, didn't you?

About the log files: Also have a look at the other logfiles, maybe they show something (the culprit might not be ssh itself).

Hope this helps.

stevena · 05-31-2010, 01:31 PM

Quote:

Originally Posted by druuna

Hi,

You don't mention it, but you did restart the ssh daemon after changing the sshd_config file, didn't you?

About the log files: Also have a look at the other logfiles, maybe they show something (the culprit might not be ssh itself).

Hope this helps.

Hi,
well the only difference is that on the other machine, PubkeyAuthentication was explicitly set to yes so i set it also in sshd_config of server 1, restarted but didn't solve the problem. i don't see how changing the shd_config file on a server could affect the ssh client from that server.
any idea what other deamons,services,programs might affect it?

stevena · 06-02-2010, 02:14 AM

update:

i upgraded all the openssh packages on machine 1 to the latest from redhat, still no luck.
previously i had

openssh-3.9p1-9.el4, openssh-server-3.9p1-9.el4, openssh-clients-3.9p1-9.el4, openssl-0.9.7a-43.17.el4_6.1

i upgraded them, along with the dependencies to

openssh-3.9p1-11.el4_7, openssh-3.9p1-11.el4_7, openssh-server-3.9p1-11.el4_7, openssh-clients-3.9p1-11.el4_7, openssl-0.9.7a-43.17.el4_8.5

the weird thing is that even after upgrading, when i run ssh -V it says

OpenSSH_3.9p1, OpenSSL 0.9.7a Feb 19 2003

thanks

druuna · 06-02-2010, 02:17 AM

Hi,

That still leaves my other question (post #12): Also have a look at the other logfiles, maybe they show something (the culprit might not be ssh itself).

Do look on both machine 1 and the one you try to connect to.