Select users unable to authenticate to CentOS 7 via ssh using PAM and NIS

ehereth · 07-29-2017, 04:45 PM

Select users unable to authenticate to CentOS 7 via ssh using PAM and NIS

Experts,

This is my first post here; please be kind and patient!

I have a problem that I've been researching for the last day and a half and I'm
at my wits' end. I think I have done all the requisite googling to no avail.

I am trying to provide a new user access to a particular server via ssh using
PAM and NIS and no matter what I try the user gets:

Quote:

"Connection to <hostname> closed by remote host"

This user is a valid user, their account is active and I can use it to access other systems on our network.

Here's what I think seems to set this problem apart from others like it that I've come across during my attempts to fix it: many other users, including myself, can access this server using the same mechanisms without any trouble. This seems to indicate to me that the ssh, PAM, NIS, etc.setting on the server are correct. So, it seems that it should be a user specific configuration, but I need help figuring out what it is.

Here is what I've done/tried:

Checked the permissions of the users home and .ssh directories as well as all
the files included in the .ssh directory. Here's what I see:

Code:

ls -l /home/ | grep user
drwx--x---   30 <user> <group> 8192 Jul 29 14:13 <user>

ls -la /home/user | grep .ssh
drwx------   2 <user> <group> 8192 Jul 29 14:13 .ssh

ls -l /home/user/.ssh/
total 56
-rw------- 1 <user> <group> 1006 Jul 29 14:04 authorized_keys
-rw------- 1 <user> <group>   31 Jul 29 14:13 config
-rw------- 1 <user> <group>  668 Jul 29 14:04 id_dsa
-rw------- 1 <user> <group>  607 Jul 29 14:04 id_dsa.pub
-rw------- 1 <user> <group> 1675 Jul 29 14:01 id_rsa
-rw------- 1 <user> <group>  399 Jul 29 14:01 id_rsa.pub
-rw------- 1 <user> <group>  444 Jul 29 14:03 known_hosts

These permissions match what I've read they should be as well as the permissions of other users who are able to connect. Furthermore, selinux is not running either.

Here is what I see when I attempt to log in the the server as the 'broken' user:

Code:

ssh -vvv remotehost
OpenSSH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /home/username/.ssh/config
debug1: /home/username/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "<remotehost>" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to <remotehost> [abc.def.ghi.jkl] port 22.
debug1: Connection established.
debug1: identity file /home/username/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/username/.ssh/id_rsa-cert type -1
debug1: identity file /home/username/.ssh/id_dsa type 2
debug1: key_load_public: No such file or directory
debug1: identity file /home/username/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/username/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/username/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/username/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/username/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1
debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to <remotehost>:22 as 'username'
debug3: hostkeys_foreach: reading file "/home/username/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/username/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from <remotehost>
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: MACs ctos: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: MACs stoc: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:SyaqgzQKI3TS7y/UOCTLLDIdojDpnl/IGPQwjk3/GYg
debug3: hostkeys_foreach: reading file "/home/username/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/username/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from <remotehost>
debug3: hostkeys_foreach: reading file "/home/username/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/username/.ssh/known_hosts:2
debug3: load_hostkeys: loaded 1 keys from abc.def.ghi.jkl
debug1: Host '<remotehost>' is known and matches the ECDSA host key.
debug1: Found key in /home/username/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS received
debug2: key: /home/username/.ssh/id_rsa (0x55945063e3e0)
debug1: Skipping ssh-dss key /home/username/.ssh/id_dsa - not in PubkeyAcceptedKeyTypes
debug2: key: /home/username/.ssh/id_ecdsa ((nil))
debug2: key: /home/username/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password,hostbased
debug3: start over, passed a different list publickey,password,hostbased
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/username/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug2: input_userauth_pk_ok: fp SHA256:/OVFW/zoG4rK0KC2aN0kVK+j12yMYZFNG6lc/7frE2k
debug3: sign_and_send_pubkey: RSA SHA256:/OVFW/zoG4rK0KC2aN0kVK+j12yMYZFNG6lc/7frE2k
debug3: send packet: type 50
debug1: Authentication succeeded (publickey).
Authenticated to <remotehost> ([abc.def.ghi.jkl]:22).
debug2: fd 5 setting O_NONBLOCK
debug3: fd 6 is O_NONBLOCK
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: send packet: type 1
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
  #0 client-session (t3 r-1 i0/0 o0/0 fd 4/5 cc -1)

debug1: fd 1 clearing O_NONBLOCK
debug3: fd 2 is not O_NONBLOCK
Connection to <remotehost> closed by remote host.
Connection to <remotehost> closed.
Transferred: sent 2580, received 2300 bytes, in 0.0 seconds
Bytes per second: sent 31006602.6, received 27641545.0
debug1: Exit status -1

And, this is what I see when I successfully ssh to the same host as myself

Code:

ssh -vvv remotehost
OpenSSH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /home/username/.ssh/config
debug1: /home/username/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "<remotehost>" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to <remotehost> [abc.def.ghi.jkl] port 22.
debug1: Connection established.
debug1: identity file /home/username/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/username/.ssh/id_rsa-cert type -1
debug1: identity file /home/username/.ssh/id_dsa type 2
debug1: key_load_public: No such file or directory
debug1: identity file /home/username/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/username/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/username/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/username/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/username/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1
debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to <remotehost>:22 as 'username'
debug3: hostkeys_foreach: reading file "/home/username/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/username/.ssh/known_hosts:643
debug3: record_hostkey: found key type RSA in file /home/username/.ssh/known_hosts:760
debug3: load_hostkeys: loaded 2 keys from <remotehost>
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-ed25519-cert-v01@openssh.com,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: MACs ctos: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: MACs stoc: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:SyaqgzQKI3TS7y/UOCTLLDIdojDpnl/IGPQwjk3/GYg
debug3: hostkeys_foreach: reading file "/home/username/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/username/.ssh/known_hosts:643
debug3: record_hostkey: found key type RSA in file /home/username/.ssh/known_hosts:760
debug3: load_hostkeys: loaded 2 keys from <remotehost>
debug3: hostkeys_foreach: reading file "/home/username/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/username/.ssh/known_hosts:644
debug3: record_hostkey: found key type RSA in file /home/username/.ssh/known_hosts:760
debug3: load_hostkeys: loaded 2 keys from abc.def.ghi.jkl
debug1: Host '<remotehost>' is known and matches the ECDSA host key.
debug1: Found key in /home/username/.ssh/known_hosts:643
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS received
debug2: key: /home/username/.ssh/id_rsa (0x55b766325930)
debug1: Skipping ssh-dss key /home/username/.ssh/id_dsa - not in PubkeyAcceptedKeyTypes
debug2: key: /home/username/.ssh/id_ecdsa ((nil))
debug2: key: /home/username/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password,hostbased
debug3: start over, passed a different list publickey,password,hostbased
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/username/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug2: input_userauth_pk_ok: fp SHA256:nNtxhdOPj8Cixs3XJEQztfsX6+UPxephCGH7of64dFo
debug3: sign_and_send_pubkey: RSA SHA256:nNtxhdOPj8Cixs3XJEQztfsX6+UPxephCGH7of64dFo
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to <remotehost> ([abc.def.ghi.jkl]:22).
debug2: fd 5 setting O_NONBLOCK
debug3: fd 6 is O_NONBLOCK
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 91
debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x10
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug3: send packet: type 98
debug1: Sending environment.
debug3: Ignored env MODULE_VERSION_STACK
debug3: Ignored env XDG_SESSION_ID
debug3: Ignored env CDLMD_LICENSE_FILE
debug3: Ignored env SHELL
debug3: Ignored env TERM
debug3: Ignored env SSH_CLIENT
debug3: Ignored env DERBY_HOME
debug3: Ignored env SSH_TTY
debug3: Ignored env USER
debug3: Ignored env TMUX
debug3: Ignored env MAIL
debug3: Ignored env PATH
debug3: Ignored env MODULE_VERSION
debug3: Ignored env PWD
debug3: Ignored env JAVA_HOME
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env MODULEPATH
debug3: Ignored env LOADEDMODULES
debug3: Ignored env TMUX_PANE
debug3: Ignored env HOME
debug3: Ignored env SHLVL
debug3: Ignored env LOGNAME
debug3: Ignored env J2SDKDIR
debug3: Ignored env SSH_CONNECTION
debug3: Ignored env TMUX_PLUGIN_MANAGER_PATH
debug3: Ignored env MODULESHOME
debug3: Ignored env XDG_RUNTIME_DIR
debug3: Ignored env J2REDIR
debug3: Ignored env BASH_FUNC_module%%
debug3: Ignored env OLDPWD
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
-bash-4.2$

In both cases, the ssh logs seem to indicate that the authentication is successful, but the other user gets kicked off immediately. As I examine logs, in particular, /var/log/secure on the remote server I see that PAM denies the user access:

Code:

Jul 29 13:14:48 <hostname> sshd[49088]: fatal: Access denied for user <user> by PAM account configuration [preauth]

In an attempt to make PAM more informative, I added 'debug' at the end of every line of the password-auth, system-auth, and postlogin files in the /etc/pam.d/ directory. This is probably overkill, but I wanted to turn no rock unturned. This didn't seem to be very helpful however; below I'll post the relevant bits from /var/log/secure for both a successful and unsuccessful login.

Here is a unsuccessful attempt:

Code:

Jul 29 16:40:10 <hostname> sshd[96747]: pam_localuser(sshd:account): checking "bin:x:1:1:bin:/bin:/sbin/nologin#012"
...
Jul 29 16:40:10 <hostname> sshd[96747]: pam_localuser(sshd:account): checking "sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin#012"
...
Jul 29 16:40:10 <hostname> sshd[96747]: pam_localuser(sshd:account): checking "ntp:x:38:38::/etc/ntp:/sbin/nologin#012"
...
Jul 29 16:40:10 <hostname> sshd[96747]: pam_localuser(sshd:account): checking "ldap:x:416:55::/var/lib/ldap:/bin/false#012"
Jul 29 16:40:10 <hostname> sshd[96747]: pam_succeed_if(sshd:account): 'uid' resolves to '<uid>'
Jul 29 16:40:10 <hostname> sshd[96747]: pam_ldap(sshd:account): nslcd authorisation; user=<user>
Jul 29 16:40:10 <hostname> sshd[96747]: pam_ldap(sshd:account): error opening connection to nslcd: No such file or directory
Jul 29 16:40:10 <hostname> sshd[96747]: fatal: Access denied for user <user> by PAM account configuration [preauth]

And here is a successful login:

Code:

Jul 29 16:51:45 <hostname> sshd[98932]: pam_localuser(sshd:account): checking "bin:x:1:1:bin:/bin:/sbin/nologin#012"
...
Jul 29 16:51:45 <hostname> sshd[98932]: pam_localuser(sshd:account): checking "sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin#012"
...
Jul 29 16:51:45 <hostname> sshd[98932]: pam_localuser(sshd:account): checking "ntp:x:38:38::/etc/ntp:/sbin/nologin#012"
...
Jul 29 16:51:45 <hostname> sshd[98932]: pam_localuser(sshd:account): checking "ldap:x:416:55::/var/lib/ldap:/bin/false#012"
Jul 29 16:51:45 <hostname> sshd[98932]: pam_succeed_if(sshd:account): 'uid' resolves to '<uid>'
Jul 29 16:51:45 <hostname> sshd[98932]: Accepted publickey for <user> from abc.def.ghi.jkl port 56522 ssh2: RSA 89:fc:b1:d5:83:68:4e:db:f0:b2:f7:59:45:b5:55:fe
Jul 29 16:51:45 <hostname> sshd[98932]: pam_keyinit(sshd:session): OPEN 2
Jul 29 16:51:45 <hostname> sshd[98932]: pam_limits(sshd:session): reading settings from '/etc/security/limits.conf'
Jul 29 16:51:45 <hostname> sshd[98932]: pam_limits(sshd:session): process_limit: processing soft memlock unlimited for DEFAULT
Jul 29 16:51:45 <hostname> sshd[98932]: pam_limits(sshd:session): process_limit: processing hard memlock unlimited for DEFAULT
Jul 29 16:51:45 <hostname> sshd[98932]: pam_limits(sshd:session): reading settings from '/etc/security/limits.d/20-nproc.conf'
Jul 29 16:51:45 <hostname> sshd[98932]: pam_limits(sshd:session): process_limit: processing soft nproc 4096 for DEFAULT
Jul 29 16:51:45 <hostname> sshd[98932]: pam_limits(sshd:session): reading settings from '/etc/security/limits.d/91-cm-limits.conf'
Jul 29 16:51:45 <hostname> sshd[98932]: pam_limits(sshd:session): process_limit: processing - memlock unlimited for DEFAULT
Jul 29 16:51:45 <hostname> sshd[98932]: pam_limits(sshd:session): process_limit: processing - stack unlimited for DEFAULT
Jul 29 16:51:45 <hostname> sshd[98932]: pam_limits(sshd:session): process_limit: processing - nofile 65536 for DEFAULT
Jul 29 16:51:45 <hostname> sshd[98932]: pam_systemd(sshd:session): pam-systemd initializing
Jul 29 16:51:45 <hostname> sshd[98932]: pam_systemd(sshd:session): Asking logind to create session: uid=<uid> pid=98932 service=sshd type=tty class=user desktop= seat= vtnr=0 tty= display= remote=yes remote_user= remote_host=<clienthost>
Jul 29 16:51:45 <hostname> sshd[98932]: pam_systemd(sshd:session): Reply from logind: id=743 object_path=/org/freedesktop/login1/session/_3743 runtime_path=/run/user/<uid> session_fd=6 seat= vtnr=0 original_uid=<uid>
Jul 29 16:51:45 <hostname> sshd[98932]: pam_succeed_if(sshd:session): 'service' resolves to 'sshd'
Jul 29 16:51:45 <hostname> sshd[98932]: pam_unix(sshd:session): session opened for user <user> by (uid=0)
Jul 29 16:51:45 <hostname> sshd[98932]: pam_ldap(sshd:session): nslcd session open; user=<user>
Jul 29 16:51:45 <hostname> sshd[98932]: pam_ldap(sshd:session): error opening connection to nslcd: No such file or directory
Jul 29 16:51:45 <hostname> sshd[98932]: pam_succeed_if(sshd:session): 'service' resolves to 'sshd'
Jul 29 16:51:45 <hostname> sshd[98932]: pam_succeed_if(sshd:session): 'service' resolves to 'sshd'

You'll see that this users' publickey gets accepted and he is allowed on the
system. I know that there is some nslcd error that confuse this issue a little;
but notice that the same messages/errors occur for both users and one of them is allowed to log in. I need to figure this out and clean it up, but I don't
think that it's part of the problem.

Also perhaps worth noting: if I force the user to use a password to log in the verbose ssh log indicates that the authentication is successful:

Code:

ssh -vvv -o PreferredAuthentications=password remotehost
...
debug1: Next authentication method: password
<user>@<remotehost>'s password:
debug3: send packet: type 50
debug2: we sent a password packet, wait for reply
debug1: Authentication succeeded (password).
...

However, in /var/log/secure I see:

Code:

Jul 29 16:59:55 <hostname> sshd[100573]: Failed password for <user> from abc.def.ghi.jkl port 56572 ssh2
Jul 29 16:59:55 <hostname> sshd[100573]: fatal: Access denied for user <user> by PAM account configuration [preauth]

I know that this users password is correct as I'm able to use to log into other
hosts on the network.

I have also checked the passwd, shadow file on the NIS server and nothing seems out of line there.

So, that's what I've got. Please feel free to request any relevant information
that I've neglected to include; I've already vomited enough information on this
post!

Thank you all very much for your time and help.

AwesomeMachine · 07-30-2017, 04:05 PM

PAM and systemd PAM are different. It's taken some time for PAM to catch up to systemd. Are you running different distros and/or versions on the problem machine and the other clients. Let's start from there.

You can try

Code:

$ dpkg-reconfigure libpam-systemd

to see if PAM on the client will configure itself properly. I'm not positive this is the problem, but I have had considerable trouble with logins and PAM with systemd.

ehereth · 07-30-2017, 04:37 PM

Thank for the suggestion; I think that it's salient to mention that I'm able successfully log into the server as myself but this new user cannot from the same client. The client I'm using is running kubuntu 16.04. I'm a little reluctant to run this command from the client as I'm afraid it might cause me to be unable to log into the problem server anymore. Is there any risk that doing this on this client will cause problems for others who are currently able to access this problem server using the client?

ehereth · 07-30-2017, 07:36 PM

So, I have a couple things to add to this question.

First, and unsurprisingly, changing UsePAM=yes to UsePAM=no in /etc/ssh/sshd_conf causes the problem to go away and the previously disallowed user is allowed to log in. I don't like this fix but it'll do in a pinch. I'd really rather figure out what the underlying problem is and fix that.

Secondly, and much more interestingly, I just added another new user and this user is able to log into the 'problem' server. A couple things that may be notable here: the new user, who can access the server, has uid=757 and gid=500 while the older user who cannot access this server has uid=1007 and gid=4244. I have look around but cannot find anything that might cause PAM to reject users with a uid/gid above a certain threshold. Could this be part of the problem though? Also, the new user's shell is bash and the older users shell is tcsh. Although, this user cannot log in after running 'bash' to switch from tcsh to bash.

I'm just trying add more information here; any help getting this resolved would be most appreciate. I need to get this 'problem' user on this server as soon as possible!

Thanks LQ!

TheEzekielProject · 07-30-2017, 11:05 PM

Have you tried adding the username to /etc/security/access.conf?

Code:

+:username:ALL

ehereth · 07-31-2017, 10:48 AM

I just tried that and it seems to make no difference. It seems that I also need to add pam_access.so to some files in /etc/pam.d, but none that I tried seemed to do the trick. Which files in that directory need to have this added to them in order for this to take effect? It seems that if I need to add it anywhere, it should look like:

Code:

account required pam_access.so

Is this correct?

Thank you for your help.

AwesomeMachine · 08-04-2017, 01:40 AM

It would need the full path:

Code:

 account required /lib/security/pam_access.so

or something similar.

I wouldn't be particularly concerned about dpkg-reconfigure messing up the system. It just checks everything to make sure it's OK.

ehereth · 08-13-2017, 10:50 AM

I'm sorry for my silence here; I got busy.

I wanted to post the solution that I discovered, although I find it most unsatisfactory.

You may recall that I mentioned that there was a lot of noise in the logs related to misconfigured LDAP settings. (PAM was checking LDAP at some point during the authentication procedure, even though we're not currently using it; yeah, this is poorly configured but it never caused other users trouble until now) I had simply been ignoring this as it hadn't prohibited other users from accessing this system. But, what ended up fixing the problem related to this users access was commenting out every reference to LDAP in both the {password,system}-auth files in /etc/pam.d/.

Does anybody have any idea as to why this was only causing problems for this single user and nobody else? I'm happy that the problem is fixed, but I feel like there should be a more satisfactory explanation to the issue.

In any event, this issue is resolved. Thank you all for your input and willingness to help!

AwesomeMachine · 08-14-2017, 07:28 PM

I would check the groups the problem user belongs to vs the groups of other users. Something might have changed with the useradd config.