[SOLVED] SSH could not start on Ubuntu 9.13

martonstark · 07-13-2023, 08:22 AM

Hi All!

I'm new in this forum, also in Linux world.
I have an issue with Ubuntu Server 9.13 VM.
I have on this server Docker and Rancher containers.

I would like to start

Code:

sudo service ssh start

and i get the following error message:

Code:

ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled) 
Active: failed (Result: timeout) since Tue 2023-07-11 15:08:00 CEST; 1 day 21h ago 
Process: 19181 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited, status=255) 
Process: 19177 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS) 
Main PID: 19181 (code=exited, status=255)
CPU: 50ms

Jul 11 15:06:29 dev systemd[1]: Starting OpenBSD Secure Shell server...
Jul 11 15:06:30 dev sshd [19181]: Server listening on 0.0.0.0 port 2222.
Jul 11 15:06:30 dev sshd [19181]: error: Bind to port 2222 on :: failed: Address already in use. Jul 11 15:06:30 dev sshd [19181] Generating 768 bit RSA key.
Jul 11 15:06:30 dev sshd [19181]: RSA key generation complete.
Jul 11 15:08:00 dev systemd[1]: ssh.service: Start operation timed out. Terminating.
Jul 11 15:08:00 dev systemd[1]: ssh.service: Main process exited, code=exited, status=255/n/a
Jul 11 15:08:00 dev systemd[1]: Failed to start OpenBSD Secure Shell server.
Jul 11 15:08:00 dev systemd[1]: ssh.service: Unit entered failed state.
Jul 11 15:08:00 dev systemd[1]: ssh.service: Failed with result 'timeout'.

What i already tried:
- changed the port to different port numbers. None of them worked.
- i used this command to check which port is listening:

Code:

netstat -tunlp

. But port 2222 is not include in the list.
- I checked all the container ports, but none of them using this port.
- restarted the whole server, nothing changed

Any idea what is the problem could be?
Should i reinstall the whole ssh and sshd service?

Thank you!

thinknix · 07-13-2023, 08:48 AM

Something is listening on port 2222. An easier way to see what that might be is with lsof:

Code:

sudo lsof -i 4:2222 -an

Here the '4' is for ipv4, use '6' for ipv6. It needs to run as root as lsof by default shows only processes owned by the user running it.

But ideally you should leave SSH on port 22, the default. There are risks involved in running SSH on an unprivileged port (>= 1024).

pan64 · 07-13-2023, 09:21 AM

just a remark, there is no ubuntu 9.13, I don't know what is it.
Yes, you need to check why is it already in use. Probably you tried to start that service two times somehow.

martonstark · 07-13-2023, 09:27 AM

Quote:

Originally Posted by thinknix

Something is listening on port 2222. An easier way to see what that might be is with lsof:

Code:

sudo lsof -i 4:2222 -an

Here the '4' is for ipv4, use '6' for ipv6. It needs to run as root as lsof by default shows only processes owned by the user running it.

But ideally you should leave SSH on port 22, the default. There are risks involved in running SSH on an unprivileged port (>= 1024).

Hi, thank you for reply.
I ran the command but no result displayed.
I set back to port 22 but the error is the same.

martonstark · 07-13-2023, 09:29 AM

Quote:

Originally Posted by pan64

just a remark, there is no ubuntu 9.13, I don't know what is it.
Yes, you need to check why is it already in use. Probably you tried to start that service two times somehow.

Hi!
Thank you, i ran this command:

Code:

lsb_release -a

And the result at the release is 9.13

thinknix · 07-13-2023, 09:37 AM

Perhaps instead of 'start' try 'restart'? What does a 'ps ax| grep sshd' show you?

michaelk · 07-13-2023, 09:58 AM

Please post the entire output of the lsb_release -a command.

Ubuntu 9.10 was released in 2009 but Ubuntu did not start using systemd until version 16.

debian's final update for version 9 was 9.13 I believe.

Ladowny · 07-13-2023, 11:24 AM

Ubunru 9 seems to be quite old. According to https://wiki.ubuntu.com/Releases it's End of Life as of April 30, 2011. Are you sure it's Ubuntu ? You would be running 12 years old OS

Don't use Ubuntu myself. however it's a fork of Debian and I run into similar problem on Debian a couple of years ago. Could have been Debian 9 at the time.

Test your ssh configuration with

Code:

sshd -t

from man sshd

Quote:

-T Extended test mode. Check the validity of the configuration file, output the effective configuration to stdout and then exit. Optionally, Match rules may be applied by
specifying the connection parameters using one or more -C options.

-t Test mode. Only check the validity of the configuration file and sanity of the keys. This is useful for updating sshd reliably as configuration options may change.

In my case I was getting "Missing privilege separation directory: /var/run/sshd"

The below is from my notes taken about 6 years ago:

create that missing directory, and try above start-stop-daemon command again:

Code:

mkdir /var/run/sshd
chmod 0755 /var/run/sshd

directory /run is ephemeral and destroyed on each reboot, so this needs to be recreated

Code:

vi /usr/lib/tmpfiles.d/sshd.conf

Make sure content is as follows ( for some reason it will not create in /var/run which is a symlink to /run )

Code:

d /run/sshd 0755 root root

martonstark · 07-14-2023, 07:08 AM

Quote:

Originally Posted by thinknix

Perhaps instead of 'start' try 'restart'? What does a 'ps ax| grep sshd' show you?

Yes i tried to restart too. Result is the same.

But for your code i get this result:

Code:

6553 ? Ss 0:00 runsv sshd

6560 7 B 0:00 svlogd -tt /var/log/gitlab/sshd

10303 7 s 0:00 /usr/shin/sshd

17109 tty1 5+ 0:00 grep --color=auto —-exclude-dir=.bzr —-exclude-dir=Cvs --exclude-dir=.gi
—-exclude-dir=.hg --exclude-dir=.svn —-exclude-dir=.idea --exclude-dir=.tox sshd

17419 7 B 1:39 tail --follow=name --retry /var/log/gitlab/gitlab-rails/api_json.log /var
|/ 10g/git1absgitlab-rails/importer. log /var/log/gitlab/gitlab-rails/application_json.log /var/log/git
lab/gitlab-rails/product ion. log /var/log/gitlab/gitlab-rails/product ion_json.log /var/log/gitlab/git
lab-rails/sidekig_client.log /var/log/gitlabsgitlab-rails/graphol_json. log /var/log/gitlab/gitlab-ra
i 15/database_load_balancing. log /var/log/gitlab/gitlab-rails/except ions_json.log /var/log/gitlab/git
lab-rails/apnlication. log /var/log/gitlah/gitlah-rails/grpc. log /var/log/gitlab/gitlab-rails/gitlab-
rails-db-migrate-2022-08-04-16-17-25. log /var/log/gitlab/gitlab-rails/auth. log /var/log/gitlab/gitle
o-rails/audit_json. log /varslog/gitlab/gitlab-rails/git_json.log /varslog/gitlab/gitlab-rails/servic
[e_measurement . log svar/log/gitlab/alertmanager/current /var/log/gitlab/alertmanagers/state /var/log/e
it 1ab/git lab-uorkhorse/current /vars/log/gitlab/gitlab-uorkhorsesstate svar/log/gitlab/postgres-expor
ter/current /var/log/gitlab/posteres-exporter/state /var/log/gitlab/registry/current /var/log/gitlat
/registru/state svarslog/gitlab/redis-exporter/current svar/log/gitlab/redis-exporter/state /var/log
/g1t 1ab/sshd/current /var/log/gitlab/gitaly/current /var/log/gitlabs/gitalys/gitaly_ruby_json. log /var
|/ 1og/git1absgitalysstate svarslog/gitlab/gitaly/gitaly_hooks. log /varslog/gitlab/sidekin/current sva
r/10g/gitlab/sidek igsstate svar/log/gitlab/prometheus/current svar/log/gitlab/prometheus/state /vars
L0g/g11 Lab/puna/puna_stdout . Log /var/1og/git lab/puma/current /varslog/gitlab/puma/puma_stderr. log /v
ar/ 102/ it 1ab/pumasstate /var/log/gitlab/redis/current var/log/gitlab/redis/state /var/log/gitlab/e
it lab-exporterscurrent svar/log/gitlab/gitlab-exporter/state /varslog/gitlab/logrotatescurrent /var/
log/gitlab/gitlab-kas/current /var/log/gitlab/gitlab-kass/state svar/log/gitlab/grafanascurrent /var/
log/gitlab/grafana/state /var/log/gitlab/nginx/current /var/logsgitlab/nging/gitlab_access. log /var/
Log/gitlab/ng inx/gitlab_registry_access. log /var/log/gitlab/nginx/error. log /var/10g/gitlab/nging/gi
[t 1ab_registry_error. log /var/log/gitlab/nginx/access. log /var/log/gitlab/nginx/gitlab_error. log /var
|/ 10g/git lab/postgresol/current /var/log/gitlab/postgresol/state

23170 7 B 0:00 sshd: /usr/shinssshd -D -f /assets/sshd_canfig -e[listener] 0 of 100-200
startups

martonstark · 07-14-2023, 07:13 AM

Quote:

Originally Posted by michaelk

Please post the entire output of the lsb_release -a command.

Ubuntu 9.10 was released in 2009 but Ubuntu did not start using systemd until version 16.

debian's final update for version 9 was 9.13 I believe.

You guys were right, sorry.
It's a Debian, sorry.

Distribution ID: Debian
Description: Debian GNU/Linux 9.13
Release: 9.13
Codename: stretch

martonstark · 07-14-2023, 07:38 AM

Quote:

Originally Posted by Ladowny

Ubunru 9 seems to be quite old. According to https://wiki.ubuntu.com/Releases it's End of Life as of April 30, 2011. Are you sure it's Ubuntu ? You would be running 12 years old OS

Don't use Ubuntu myself. however it's a fork of Debian and I run into similar problem on Debian a couple of years ago. Could have been Debian 9 at the time.

Test your ssh configuration with

Code:

sshd -t

from man sshd

In my case I was getting "Missing privilege separation directory: /var/run/sshd"

The below is from my notes taken about 6 years ago:

create that missing directory, and try above start-stop-daemon command again:

Code:

mkdir /var/run/sshd
chmod 0755 /var/run/sshd

directory /run is ephemeral and destroyed on each reboot, so this needs to be recreated

Code:

vi /usr/lib/tmpfiles.d/sshd.conf

Make sure content is as follows ( for some reason it will not create in /var/run which is a symlink to /run )

Code:

d /run/sshd 0755 root root

Thank you. I tried the

Code:

sshd -t

and with -T but there was no output.

Also I tried the solution what you had before, and the result is remaind almost same.

Code:

 Unit ssh.service has begun starting up.

Ul 14 14:24:5¢ dev sshd[5269]: error: Bind to port 2222 on 0.0.0.0 failed: Address already in use.
ul 14 14:24:54 dev sshd[5269]: error: Bind to port 2222 on :: failed: Address already in use.
Ul 14 14:24:54 dev systemd[1]: ssh.service: Main process exited, codesexited, status=zss/n/a
ul 14 14:24:5¢ dev sshd[5269]: fatal: Cannot bind any address.

ul 14 14:24:5¢ dev systemd[1]: Failed to start OpenBSD Secure Shell server.

- Subject: Unit ssh.service has failed

- Defined-By: systemd

- Support: https://www.debian.org/support

- Unit ssh.service has failed.

- The result is failed.

Ul 14 14:24:5¢ dev systemd[1]: ssh.service: Unit entered failed state.

Ul 14 14:24:54 dev sistemd[i]: ssh.service: Failed with result 'exit-code'.

thinknix · 07-14-2023, 08:02 AM

Quote:

Originally Posted by martonstark

Yes i tried to restart too. Result is the same.

But for your code i get this result:

Code:

6553 ? Ss 0:00 runsv sshd
10303 7 s 0:00 /usr/shin/sshd
23170 7 B 0:00 sshd: /usr/shinssshd -D -f /assets/sshd_canfig -e[listener] 0 of 100-200
startups

I trimmed your 'ps' output, you have three sshd instances already running. Also, 'runsv' is part of the runit init system, yet your logs show systemd info. This is bound to cause issues much larger than this one daemon not starting.

I'm not sure what that last sshd is: "/usr/shinssshd -D -f /assets/sshd_canfig -e[listener] 0 of 100-200" or "/usr/shin/sshd", which seems like a typo and should be '/usr/sbin/sshd'.

martonstark · 07-18-2023, 04:44 AM

Quote:

Originally Posted by thinknix

I trimmed your 'ps' output, you have three sshd instances already running. Also, 'runsv' is part of the runit init system, yet your logs show systemd info. This is bound to cause issues much larger than this one daemon not starting.

And what i could do? Should I kill this processes? I tried that but it has been shown in the list again with new process ID.

Code:

kill <process id>

and

Code:

kill -9 <process id>

Quote:

I'm not sure what that last sshd is: "/usr/shinssshd -D -f /assets/sshd_canfig -e[listener] 0 of 100-200" or "/usr/shin/sshd", which seems like a typo and should be '/usr/sbin/sshd'.

Yes this is looks like a typo, i ran again and the pathes are fine.

Thank you.

pan64 · 07-18-2023, 05:54 AM

no, killing them will not solve your issue at all. You need to find out why did they start (and how). And obviously you only need one of them, the others can be stopped/disabled/removed/fixed.

Skaperen · 07-18-2023, 08:16 PM

maybe your situation is that there is also a different problem that keeps sshd running even if it does not work as intended. if you reboot, it would start 1st sshd and fail to work as expected for a different reason while continuing to run. then starting it again is hitting the busy port problem because the 1st one is still listening on that port. ultimately, you need to first solve why the port stays busy ... what the other problem is.

what i would do is change the configuration so that sshd does not get started automatically at all. that way i only need to reboot once and can manually start sshd to test changes without a reboot each time. then once i have it working OK, change it back to automatic start and do one more reboot to be sure the fix really works on boot up.