ssh login failing for 1 user, not for another

Timothy Miller · 05-03-2017, 03:08 PM

So, I'm setting up a server that connects to several ftp sites to put files. We need multiple users in order that the files aren't able to be read by the user that uses 1 (client requirement). So I have user prod and test. Both are set up the same way. I am actually testing with the same private/public key. test can login, prod can't. prod gets the error that server refused the key. Another user (centos, this is AWS instance) also is able to login with their key just fine (different keypair to these users).

The /var/log/secure shows only

Quote:

error: Received disconnect from 10.0.0.154: 14: No supported authentication methods available [preauth]

, implying that the server can't see the authorized_keys file. I triple checked spelling and had a coworker check (since this is very common error for me).

~/.ssh/authorized_keys is set to 400 on both users
~/.ssh is set to 500 on both users
No users or groups are denied in /etc/ssh/sshd_config to login

Users local account works correctly. You can login to it via sudo su - prod and everything works like it should. The profiles (sudo -Hiu prod env vs. sudo -Hiu test env) are basically identical other than the obvious.

I'm out of ideas of what it could be, and while normally I'd just delete the user and recreate, in this case that's not an option.

Code:

sudo -Hiu prod env
HOSTNAME=ip-10-0-1-4.us-west-2.compute.internal
SHELL=/bin/bash
TERM=xterm
HISTSIZE=1000
USER=prod
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
SUDO_USER=root
SUDO_UID=0
USERNAME=prod
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/home/prod/.local/bin:/home/prod/bin
MAIL=/var/spool/mail/prod
PWD=/home/prod
LANG=en_US.UTF-8
HISTCONTROL=ignoredups
SHLVL=1
SUDO_COMMAND=/bin/bash -c env
HOME=/home/prod
LOGNAME=prod
LESSOPEN=||/usr/bin/lesspipe.sh %s
SUDO_GID=0
_=/bin/env

Code:

sudo -Hiu test env
HOSTNAME=ip-10-0-1-4.us-west-2.compute.internal
SHELL=/bin/bash
TERM=xterm
HISTSIZE=1000
USER=test
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
SUDO_USER=root
SUDO_UID=0
USERNAME=test
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/home/test/.local/bin:/home/test/bin
MAIL=/var/spool/mail/test
PWD=/home/test
LANG=en_US.UTF-8
HISTCONTROL=ignoredups
SHLVL=1
SUDO_COMMAND=/bin/bash -c env
HOME=/home/test
LOGNAME=test
LESSOPEN=||/usr/bin/lesspipe.sh %s
SUDO_GID=0
_=/bin/env

Code:

grep 'prod\|test' /etc/passwd
prod:x:1001:1001::/home/prod:/bin/bash
test:x:1002:1002::/home/test:/bin/bash

Habitual · 05-03-2017, 03:35 PM

Mr. Miller:

Couple of things, casually...

Are the contents of

Code:

/home/prod/.ssh/authorized_keys

and

Code:

/home/test/.ssh/authorized_keys

identical?

and/or
possibly

Code:

/etc/hosts.{allow,deny}

and/or
Security Group as implied by the internal IP of 10.0.0.154

and/or
does prod have a linux passwd set? (su - prod is "iffy" as test, IMO)

Why is ftp hitting /var/log/secure is my Q.
Why are keys involved with file transfer protocol?

It implies sftp but I'm punting here.
and I Live for "the obvious"

Peace.

MensaWater · 05-03-2017, 03:35 PM

Did you check the permissions of ~ (i.e. the user's home directory itself) for both users. How about directories above that?

The "s" in ssh and sftp is "secure" and they will refuse to connect if they deem the connection to be insecure. If other users can access the ssh/sftp user's home directory because of permissions on it or directories above it then it will refuse the connection just as it would if ~.ssh for the user or any of its files didn't have the correct permissions.
For many home directories the directory is "/home/<user>" so you'd have to verify "/", "/home" and "/<user>" are restricting writes by other users. Usually I find / and /home to be okay and setting 750 on /home/<user> is sufficient.

Timothy Miller · 05-03-2017, 03:42 PM

Quote:

Originally Posted by Habitual

Mr. Miller:

Couple of things, casually...

Are the contents of

Code:

/home/prod/.ssh/authorized_keys

and

Code:

/home/test/.ssh/authorized_keys

identical?

Yes, copied from /test and chowned it to prod.

Quote:

and/or
possibly

Code:

/etc/hosts.{allow,deny}

Both files empty of configuration

Quote:

and/or
Security Group as implied by the internal IP of 10.0.0.154

and/or
does prod have a linux passwd set? (su - prod is "iffy" as test, IMO)

No passwords set for any user on machine, since this is AWS, fully key-based logins and passwords are only used for sudo (which neither of these users have access to).

Quote:

Why is ftp hitting /var/log/secure is my Q.
Why are keys involved with file transfer protocol?

It implies sftp but I'm punting here.

Peace.

The users are only used to transfer the files TO the client ftp site, that part is working just fine, but the files are collected on this machine via scp from multiple other machines, and that's what's not working (well, it is for test).

Quote:

Originally Posted by MensaWater

Did you check the permissions of ~ (i.e. the user's home directory itself) for both users. How about directories above that?

Yes, multiple times

Quote:

The "s" in ssh and sftp is "secure" and they will refuse to connect if they deem the connection to be insecure. If other users can access the ssh/sftp user's home directory because of permissions on it or directories above it then it will refuse the connection just as it would if ~.ssh for the user or any of its files didn't have the correct permissions.
For many home directories the directory is "/home/<user>" so you'd have to verify "/", "/home" and "/<user>" are restricting writes by other users. Usually I find / and /home to be okay and setting 750 on /home/<user> is sufficient.

As stated, permissions on ~/.ssh on both is 500, ~/.ssh/authorized_keys is 400.

michaelk · 05-03-2017, 04:57 PM

Lets see if I understand what is happening...

You did not post anything about the client but I assume that you are using the same computer/username/private key and trying to login to the server using either prod or test as the remote server name? I also assume that password authentication is disabled?

I also assume that you created you the keys for test first which work and then copied the public key from test to prod on the server which does not work.

If using a linux client you can try adding debug information to see if that adds any additional information.

Timothy Miller · 05-03-2017, 06:06 PM

Quote:

Originally Posted by michaelk

Lets see if I understand what is happening...

You did not post anything about the client but I assume that you are using the same computer/username/private key and trying to login to the server using either prod or test as the remote server name? I also assume that password authentication is disabled?

I also assume that you created you the keys for test first which work and then copied the public key from test to prod on the server which does not work.

If using a linux client you can try adding debug information to see if that adds any additional information.

Correct on most counts, the prod & test are just the users on the remote server that I'm trying to connect to, it's the same remote server name for both users (client doesn't want prod data to be handled by test and vice versa, so even though all data is collected on the same server, and ftp'd to the same ftp server, we require 2 users to do it). Also connecting from Windows clients.

michaelk · 05-03-2017, 06:41 PM

About the only thing I can think of at the moment is mismatched keys but that does not agree with my assumptions.

Habitual · 05-03-2017, 06:52 PM

Quote:

Originally Posted by Timothy Miller

As stated, permissions on ~/.ssh on both is 500, ~/.ssh/authorized_keys is 400.

This ^^ sticks out.

On more than one AWS host, we have it

Code:

stat --printf "%a %n \n" .ssh .ssh/authorized_keys
700 .ssh 
600 .ssh/authorized_keys

and the same key used at will on any host.

Just sayin'

michaelk · 05-03-2017, 06:55 PM

Thought about that too but if it works on one username it should on the other.

Timothy Miller · 05-03-2017, 09:49 PM

I use 500/400 on all servers, as many DO have realk living users connecting to them to do various things, and this reduces the likelihood of them from doing something stupid.

MensaWater · 05-04-2017, 07:37 AM

Quote:

Originally Posted by Timothy Miller

As stated, permissions on ~/.ssh on both is 500, ~/.ssh/authorized_keys is 400.

Yes you said that before, but what are the permissions on the user's home and the directories above it? Just saying you checked them and then saying again what the permissions were on the .ssh subdirectory and its file makes me suspect you didn't understand what I was saying about the other directories.

Laserbeak · 05-04-2017, 07:49 AM

Looks like you've fully examined the server. I'd look at the client... do they have the keys set up properly? Are they using the right ssh command?

Laserbeak · 05-04-2017, 07:56 AM

Quote:

normally I'd just delete the user and recreate, in this case that's not an option.

Why not? The delete user command for your system should have the option to preserve the home directory (But, just to be super-careful, I'd tar it first too and put it in a safe place) and then you can immediately recreate the same user.

Timothy Miller · 05-04-2017, 08:09 AM

Quote:

Originally Posted by MensaWater

Yes you said that before, but what are the permissions on the user's home and the directories above it? Just saying you checked them and then saying again what the permissions were on the .ssh subdirectory and its file makes me suspect you didn't understand what I was saying about the other directories.

Oh, the home directories are 750

Quote:

Originally Posted by Laserbeak

Looks like you've fully examined the server. I'd look at the client... do they have the keys set up properly? Are they using the right ssh command?

keys are set up properly, to test I'm using the test user keypair for both and doing it from my actual workstation.

Quote:

Originally Posted by Laserbeak

Why not? The delete user command for your system should have the option to preserve the home directory (But, just to be super-careful, I'd tar it first too and put it in a safe place) and then you can immediately recreate the same user.

The directory isn't the important part (I've deleted it multiple times hoping maybe it was just a permissions/spelling issue), these users also are system accounts that software is running as. Actually, they're just system accounts that because this client wanted to do things differently than everyone else were then required to be able to have ssh (scp) working...

MensaWater · 05-04-2017, 08:39 AM

Quote:

Originally Posted by Timothy Miller

Oh, the home directories are 750

And the directories all the way back to / (root) from the homes are...?