Suddenly get "Connection refused" when SSHing to AWS EC2 instance

sllinux · 12-04-2022, 03:12 AM

We have an AWS EC2 instance running Amazon Linux 2. This instance was running fine up until a couple days ago.

Then we used an Ansible playbook to install a new version of our software. We have used this same playbook many times before without any problems. But this time, after the playbook said it successfully executed, we could no longer SSH to the instance - we get a "Connection refused".

We tried stopping the EC2 instance and restarting it from the AWS console, but we still get a "Connection refused".

It is not a security group issue, because I can SSH to another instance in the same VPC and subnet as the inaccessible instance. Both of those instances have the exact same security groups.

I have detached the EBS volume on the inaccessible instance and attached it to another EC2 instance that I am able to SSH to. I compared the /etc/ssh/sshd_config file on the volume to the /etc/ssh/sshd_config file of the instance that I can SSH to, and they are exactly the same. So the sshd_config file didn't get corrupted.

In the AWS console, when doing "Get system log" for the instance, one of the messages says "Started OpenSSH server daemon", so it looks like the SSH daemon is up and running. Also, it shows a login prompt, so the instance seems to be running - I just can't SSH into it. (I also cannot use the AWS console to login to any of our instances.)

So the only thing I can do is to detach the EBS volume from the inaccessible instance and attach it to another instance that I can SSH to. Then I can view/edit files on the volume. What else can I look at to try to figure out why I can't SSH to the instance?

evo2 · 12-04-2022, 09:34 PM

Did you check logs from sshd?

Evo2.

Turbocapitalist · 12-04-2022, 10:25 PM

+1 for checking the SSH server logs.

Also, there is a slight chance that increasing the verbosity of the client may tell something useful, depending on the problem. See the -v and -E options in "man ssh".

sllinux · 12-05-2022, 12:50 AM

Doing "ssh -vvv" gives this output:

Code:

$ ssh -vvv -i .ssh/ourkey.pem ec2-user@10.130.35.176
OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "10.130.35.176" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 10.130.35.176 [10.130.35.176] port 22.
debug1: connect to address 10.130.35.176 port 22: Connection refused
ssh: connect to host 10.130.35.176 port 22: Connection refused

The only sshd logs I can find are in /var/log/secure:

Code:

Dec  4 19:21:33 ip-10-130-35-176 sshd[3265]: Received signal 15; terminating.
Dec  4 20:10:23 ip-10-130-35-176 sshd[3280]: Server listening on 0.0.0.0 port 22.
Dec  4 20:10:23 ip-10-130-35-176 sshd[3280]: Server listening on :: port 22.

I don't see any messages from sshd in /var/log/messages or /var/log/dmesg. Anywhere else to look?

Turbocapitalist · 12-05-2022, 02:27 AM

Can you first verify that the SSH daemon really is up and running on your AWS system?

If it is then, usually the logs from sshd are in /var/log/secure or /var/log/auth depending on the distro.

Another option, if you can have another port open than 22 would be to start up a single-session daemon, saving the logs to a separate file.

Code:

sudo /usr/sbin/sshd -p 2222 -d -E /tmp/sshd.log
sudo /usr/sbin/sshd -p 2222 -d -d -E /tmp/sshd.log
sudo /usr/sbin/sshd -p 2222 -d -d -d -E /tmp/sshd.log

That will allow you connect once, and only once, with SSH on port 2222 and store all the connection logs in the file /tmp/sshd.log It will otherwise use the settings from the main configuration file but won't conflict with the existing server on port 22.

sllinux · 12-05-2022, 09:22 AM

I'm not sure how else to tell if the SSH daemon is really up and running besides seeing the "Started OpenSSH server daemon" in the system log. There is only the /var/log/secure on this system, no /var/log/auth.

I added "@reboot sudo /usr/sbin/sshd -p 2222 -d -E /tmp/sshd.log" to the /var/spool/cron/root file, added an AWS security group rule to allow TCP on port 2222, then started up the system again.

SSH got the same error as before:

Code:

$ ssh -p 2222 -vvv -i .ssh/ourkey.pem ec2-user@10.130.35.176
OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "10.130.35.176" port 2222
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 10.130.35.176 [10.130.35.176] port 2222.
debug1: connect to address 10.130.35.176 port 2222: Connection refused
ssh: connect to host 10.130.35.176 port 2222: Connection refused

On the server, the /tmp/sshd.log file contains:

Code:

debug1: sshd version OpenSSH_7.4, OpenSSL 1.0.2k-fips  26 Jan 2017
debug1: private host key #0: ssh-rsa SHA256:c3moWDL/m3TZDrNjam8U+wsyldUJBYIVWc/7/j4POKc
debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:Z52ONg5SMAVk64nkLpqsB5rj6ct0LIb4a63geHpk64R
debug1: private host key #2: ssh-ed25519 SHA256:lsn65R5BUiO5G4h02O0bmn8TZBxIYTVzjwQp7I5eVys
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-p'
debug1: rexec_argv[2]='2222'
debug1: rexec_argv[3]='-d'
debug1: rexec_argv[4]='-E'
debug1: rexec_argv[5]='/tmp/sshd.log'
debug1: Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port 2222 on 0.0.0.0.
Server listening on 0.0.0.0 port 2222.
debug1: Bind to port 2222 on ::.
Server listening on :: port 2222.
Received signal 15; terminating.

elgrandeperro · 12-05-2022, 10:23 AM

It isn't getting to sshd, as far as /var/log/secure. So I'd look at iptables/firewalled.

evo2 · 12-05-2022, 02:14 PM

Trying to ssh to localhost can also provide useful information. Eg

Code:

ssh -vvv localhost

Also, try manually starting sshd on a high numbered port in debug mode. Then try to ssh to that instance from the localhost.

Code:

/usr/sbin/sshd -d -p 9876

Then in another terminal

Code:

ssh -vvv -p 9876 localhost

Then check the output in both terminals.

Evo2.

sllinux · 12-17-2022, 02:00 PM

Sorry for the delay, was sidetracked by some other things. After further investigation, it appears that this issue occurs when trying to SSH to the instance while on the corporate VPN from home. I was in the office last week and had no problem SSHing to the instance.

However, it's just a problem with this one instance, because I have no problem SSHing to any of the other EC2 instances while on the corporate VPN at home. I asked our IT department about it, and they checked the corporate firewall and said nothing is blocking SSH traffic from the corporate VPN to that instance.

I ran "sudo tcpdump -i eth0 src <my VPN IP address> and port 22", and confirmed that my SSH request from the corporate VPN is making it to the instance.

So then is it something with iptables? Here are the iptables rules on the instance:

Code:

$ sudo iptables -S
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o br-b6f99425f27e -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-b6f99425f27e -j DOCKER
-A FORWARD -i br-b6f99425f27e ! -o br-b6f99425f27e -j ACCEPT
-A FORWARD -i br-b6f99425f27e -o br-b6f99425f27e -j ACCEPT
-A DOCKER -d 172.30.0.3/32 ! -i br-b6f99425f27e -o br-b6f99425f27e -p tcp -m tcp --dport 41414 -j ACCEPT
-A DOCKER -d 172.30.0.3/32 ! -i br-b6f99425f27e -o br-b6f99425f27e -p tcp -m tcp --dport 4545 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-b6f99425f27e ! -o br-b6f99425f27e -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-b6f99425f27e -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN