My Server got Hacked?

hannso · 10-08-2023, 10:16 AM

Dear all,
I am running out of ideas and need inspiration how to track down my current security issue.

Why I think that my server got hacked?

From time to time, when I execute this command

Code:

sudo lsof -i -P -n | grep ssh

I get connection as user root to some ip in China/US/xxx,
and I know, that I am the only admin of that server.

Code:

COMMAND     PID            USER   FD   TYPE     DEVICE SIZE/OFF NODE NAME
sshd      15373            root    3u  IPv4 3237744941      0t0  TCP xx.xxx.211.184:23->110.85.99.146:35587 (ESTABLISHED)

The connection does not live for a long time and is reestablished once a minute or so.

What I already tried?

- Moving ssh to port 23
- ufw is up and running
- checked sshd_config which is

Quote:

Port 23
PermitRootLogin no
PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM no
PrintMotd no
Subsystem sftp /usr/lib/openssh/sftp-server

- checked the pid

Quote:

ps -Flww -p 15786
F S UID PID PPID C PRI NI ADDR SZ WCHAN RSS PSR STIME TTY TIME CMD
4 S root 15786 15714 0 80 0 - 18076 - 5692 1 17:08 ? 00:00:00 sshd: [accepted]

- Checked its status via cat /proc/<pid>/status

Quote:

cat /proc/16022/status
Name: sshd
Umask: 0022
State: S (sleeping)
Tgid: 16022
Ngid: 0
Pid: 16022
PPid: 15714
TracerPid: 0
Uid: 0 0 0 0
Gid: 0 0 0 0
FDSize: 64
Groups:
NStgid: 16022
NSpid: 16022
NSpgid: 16022
NSsid: 16022
VmPeak: 72336 kB
VmSize: 72304 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 5716 kB
VmRSS: 5716 kB
RssAnon: 732 kB
RssFile: 4984 kB
RssShmem: 0 kB
VmData: 760 kB
VmStk: 132 kB
VmExe: 756 kB
VmLib: 8964 kB
VmPTE: 180 kB
VmSwap: 0 kB
HugetlbPages: 0 kB
CoreDumping: 0
Threads: 1
SigQ: 0/15277
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000001000
SigCgt: 0000000180002000
CapInh: 0000000000000000
CapPrm: 0000003fffffffff
CapEff: 0000003fffffffff
CapBnd: 0000003fffffffff
CapAmb: 0000000000000000
NoNewPrivs: 0
Seccomp: 0
Speculation_Store_Bypass: thread vulnerable
Cpus_allowed: 3
Cpus_allowed_list: 0-1
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,0 0000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00 000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 1
nonvoluntary_ctxt_switches: 0

- installed and ran rkhunter

What I could not check

Don't know if a docker container was hacked and this sshd is runnin in there

Thank you very much for your ideas!

Petri Kaukasoina · 10-08-2023, 02:39 PM

Quote:

Originally Posted by hannso

I get connection as user root to some ip in China/US/xxx,

Let's rephrase it: sshd runs as root (as it should) and some script tries to connect it from some ip in China/US. It goes through a dictionary trying one username and one password at a time. It does not get access, and it tries again once a minute or so. Every machine connected to the internet sees this unless prohibited by a firewall. No problem.

Keith Hedger · 10-08-2023, 03:31 PM

install fail2ban
https://www.linuxquestions.org/quest...ux-4175729669/

hannso · 10-08-2023, 04:33 PM

Quote:

Originally Posted by Petri Kaukasoina

Let's rephrase it: sshd runs as root (as it should) and some script tries to connect it from some ip in China/US. It goes through a dictionary trying one username and one password at a time. It does not get access, and it tries again once a minute or so. Every machine connected to the internet sees this unless prohibited by a firewall. No problem.

Thank you very much!

Is your hypothesis also true if the connection remains open for several seconds? (~20)
(using the lsof command)

michaelk · 10-08-2023, 05:27 PM

The default is 6 tries at entering a password. Someone calculated the time per connection at 19 seconds. Turning off password authentication is good but ssh still prompts for a password if a key was not sent. The script kiddies don't know that and continue to try for awhile and move on to the next IP address. Changing the number of tries helps including using fail2ban but does not prevent them from constantly trying.

https://superuser.com/questions/1148...mpt-rate-limit

jayjwa · 10-08-2023, 10:00 PM

Don't run anything on port 23 open to the net at large because it gets harassed too much. There's alot of worms that hit that port. Whatever you put there will have to talk to each worm and hacker that connects. If you have telnet open on LAN, rope it off with libwrap. As for ssh, move it to another high port. Enter 'secret-ssh' in /etc/services on your chosen port. Make use of Xinetd's sensor function to auto-ban IPs for some set time. If they trip that sensor, you don't want them playing with your other services, right? You'll still see a few attempts, but it will only be a couple a day. Make sure you don't hit the port yourself.

xinetd.conf

Code:

service secret-ssh
{
        flags           = IPv6
        instances       = 10
        socket_type     = stream
        protocol        = tcp
        wait            = no
        user            = root
        server          = /usr/sbin/sshd
        server_args     = -i
}

service ssh
{
        flags           = SENSOR
        type            = INTERNAL
        socket_type     = stream
        wait            = no
        user            = nobody
        deny_time       = 45
}

23/10/8@05:19:07 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 94.102.61.20 to the global_no_access list for 45 minutes
23/10/8@05:31:10 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 200.2.170.187 to the global_no_access list for 45 minutes
23/10/8@05:37:10 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 220.134.47.143 to the global_no_access list for 45 minutes
23/10/8@05:47:48 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 175.193.76.97 to the global_no_access list for 45 minutes
23/10/8@06:05:39 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 222.120.245.161 to the global_no_access list for 45 minutes
23/10/8@06:09:42 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 24.155.93.133 to the global_no_access list for 45 minutes
23/10/8@06:28:21 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 178.62.237.183 to the global_no_access list for 45 minutes
23/10/8@06:36:46 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 167.94.146.53 to the global_no_access list for 45 minutes
23/10/8@06:39:59 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 87.236.176.247 to the global_no_access list for 45 minutes
23/10/8@06:39:59 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 188.166.26.88 to the global_no_access list for 45 minutes
23/10/8@06:48:57 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 221.158.77.170 to the global_no_access list for 45 minutes
23/10/8@06:54:56 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 198.235.24.94 to the global_no_access list for 45 minutes
23/10/8@06:58:07 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 120.50.75.204 to the global_no_access list for 45 minutes
23/10/8@07:23:26 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 194.26.29.16 to the global_no_access list for 45 minutes
23/10/8@07:25:08 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 171.227.35.196 to the global_no_access list for 45 minutes
23/10/8@07:30:24 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 146.190.60.250 to the global_no_access list for 45 minutes
23/10/8@08:18:25 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 198.235.24.146 to the global_no_access list for 45 minutes
23/10/8@08:23:14 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 74.93.12.74 to the global_no_access list for 45 minutes
23/10/8@08:42:00 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 49.163.9.237 to the global_no_access list for 45 minutes
23/10/8@08:46:11 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 157.245.98.245 to the global_no_access list for 45 minutes
23/10/8@08:55:56 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 218.144.194.30 to the global_no_access list for 45 minutes
23/10/8@09:08:56 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 87.4.246.56 to the global_no_access list for 45 minutes
23/10/8@09:14:25 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 167.94.145.59 to the global_no_access list for 45 minutes
23/10/8@09:21:24 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 59.23.217.197 to the global_no_access list for 45 minutes
23/10/8@09:33:30 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 167.94.146.51 to the global_no_access list for 45 minutes
23/10/8@09:54:16 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 118.34.209.108 to the global_no_access list for 45 minutes
23/10/8@09:59:26 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 44.192.19.231 to the global_no_access list for 45 minutes
23/10/8@10:56:07 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 65.49.1.29 to the global_no_access list for 45 minutes
23/10/8@11:06:31 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 104.236.194.175 to the global_no_access list for 45 minutes
23/10/8@11:07:20 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 85.209.11.227 to the global_no_access list for 45 minutes
23/10/8@11:47:37 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 104.152.52.87 to the global_no_access list for 45 minutes
23/10/8@12:36:58 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 49.161.211.214 to the global_no_access list for 45 minutes
23/10/8@12:40:51 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 112.173.196.53 to the global_no_access list for 45 minutes
23/10/8@12:45:06 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 198.235.24.239 to the global_no_access list for 45 minutes
23/10/8@13:00:32 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 141.98.11.116 to the global_no_access list for 45 minutes
23/10/8@13:08:44 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 103.147.34.150 to the global_no_access list for 45 minutes
23/10/8@13:13:14 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 47.74.96.31 to the global_no_access list for 45 minutes
23/10/8@13:35:29 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 36.232.160.80 to the global_no_access list for 45 minutes
23/10/8@13:44:03 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 196.219.210.179 to the global_no_access list for 45 minutes
23/10/8@13:57:20 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 71.31.148.53 to the global_no_access list for 45 minutes
23/10/8@13:58:37 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 79.6.222.21 to the global_no_access list for 45 minutes
23/10/8@14:00:51 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 122.116.63.228 to the global_no_access list for 45 minutes
23/10/8@14:03:21 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 103.145.5.200 to the global_no_access list for 45 minutes
23/10/8@14:13:01 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 73.5.43.53 to the global_no_access list for 45 minutes
23/10/8@14:26:23 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 177.135.83.2 to the global_no_access list for 45 minutes
23/10/8@14:36:08 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 162.240.38.128 to the global_no_access list for 45 minutes
23/10/8@14:58:43 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 121.157.153.45 to the global_no_access list for 45 minutes
23/10/8@15:13:03 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 47.74.96.31 to the global_no_access list for 45 minutes
23/10/8@15:19:18 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 175.198.154.67 to the global_no_access list for 45 minutes
23/10/8@15:53:12 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 170.64.212.209 to the global_no_access list for 45 minutes
23/10/8@16:00:15 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 220.136.12.42 to the global_no_access list for 45 minutes
23/10/8@16:10:57 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 194.26.29.16 to the global_no_access list for 45 minutes
23/10/8@16:12:55 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 183.109.240.16 to the global_no_access list for 45 minutes
23/10/8@16:27:36 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 94.183.87.237 to the global_no_access list for 45 minutes
23/10/8@16:33:35 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 203.136.74.88 to the global_no_access list for 45 minutes
23/10/8@17:11:28 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 165.227.201.141 to the global_no_access list for 45 minutes
23/10/8@18:24:26 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 192.254.222.170 to the global_no_access list for 45 minutes
23/10/8@18:25:26 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 221.157.12.243 to the global_no_access list for 45 minutes
23/10/8@18:42:05 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 167.248.133.37 to the global_no_access list for 45 minutes
23/10/8@18:58:32 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 198.235.24.95 to the global_no_access list for 45 minutes
23/10/8@19:05:40 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 222.114.245.223 to the global_no_access list for 45 minutes
23/10/8@19:10:49 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 167.71.68.224 to the global_no_access list for 45 minutes
23/10/8@19:14:52 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 141.98.11.116 to the global_no_access list for 45 minutes
23/10/8@19:34:43 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 34.236.192.45 to the global_no_access list for 45 minutes
23/10/8@19:44:48 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 175.197.192.209 to the global_no_access list for 45 minutes
23/10/8@19:56:23 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 146.190.152.77 to the global_no_access list for 45 minutes
23/10/8@20:03:22 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 125.191.29.82 to the global_no_access list for 45 minutes
23/10/8@20:05:06 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 205.210.31.225 to the global_no_access list for 45 minutes
23/10/8@20:14:22 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 95.229.149.224 to the global_no_access list for 45 minutes
23/10/8@20:22:22 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 218.146.21.133 to the global_no_access list for 45 minutes
23/10/8@20:57:09 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 121.182.170.71 to the global_no_access list for 45 minutes
23/10/8@21:07:19 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 46.252.16.96 to the global_no_access list for 45 minutes
23/10/8@21:57:45 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 180.183.64.212 to the global_no_access list for 45 minutes
23/10/8@22:32:54 xinetd[1148]: CRITICAL: 1148 {process_sensor} Adding 192.241.226.52 to the global_no_access list for 45 minutes

michaelk · 10-08-2023, 10:24 PM

The ban will be just for port 22, the script kiddies will probe for ssh so just running it on another port isn't much of a deterrent.

replica9000 · 10-09-2023, 10:09 AM

I run sshd on a much higher port. While it will occasionally gets some attacks, it's far less than using something within the standard 0-1023 range.

With sshd, not only can you configure it not to allow root login, but can use the 'Match user' option to configure who can login and what they can do.

fail2ban is a good option as well. It'll ban attackers after X amount of failed attempts for the duration you choose, or permanently. It'll also send you a detailed email when an attacker is banned.

For me, 95% of the time, someone will try several common usernames, get the first ban and give up.

jayjwa · 10-09-2023, 12:28 PM

Quote:

The ban will be just for port 22

No, it's every service under xinetd. Here, QotD and fake ssh I ban myself (just restart xinetd to clear).

Code:

jayjwa@atr2 ~ [SIGINT]> ncat -v 74.a.b.c 17
Ncat: Version 7.94 ( https://nmap.org/ncat )
Ncat: Connected to 74.a.b.c:17.
I would like to urinate in an OVULAR, porcelain pool --

^C⏎                                                                             
jayjwa@atr2 ~ [SIGINT]> ncat -v 74.a.b.c 22
Ncat: Version 7.94 ( https://nmap.org/ncat )
Ncat: Connected to 74.a.b.c:22.

^C⏎                                                                            
23/10/9@13:09:34 xinetd[22448]: CRITICAL: 22448 {process_sensor} Adding 74.a.b.c to the global_no_access list for 45 minutes

jayjwa@atr2 ~ [SIGINT]> ncat -v 74.a.b.c 17
Ncat: Version 7.94 ( https://nmap.org/ncat )
Ncat: Connected to 74.a.b.c:17.

^C⏎

xinetd.log

Code:

23/10/9@13:09:11: START: qotd-stream pid=22461 from=::ffff:74.a.b.c
23/10/9@13:09:11: EXIT: qotd-stream status=0 pid=22461 duration=0(sec)
23/10/9@13:09:34: FAIL: ssh address from=74.a.b.c
23/10/9@13:09:42: START: qotd-stream pid=22505 from=::ffff:74.a.b.c
23/10/9@13:09:42: FAIL: qotd-stream address from=::ffff:74.a.b.c
23/10/9@13:09:42: EXIT: qotd-stream status=0 pid=22505 duration=0(sec)

sundialsvcs · 10-09-2023, 06:55 PM

I've said it before and I'll say it again. You can have "Number of unauthorized access attempts: Zero" if you put digital certificate based OpenVPN with 'tls-auth' as your outermost line of defense. Your "sshd" then listens only to the virtual internal network which represents clients who have successfully passed through the VPN.

'tls-auth' basically hides the fact that an OpenVPN server is even out there. No ports are open, except maybe a web server. Authorized users with non-revoked certificates can pass over the (hidden ...) moat quite easily and reach the portcullis. No one else can even see that it exists.

Keith Hedger · 10-10-2023, 08:22 AM

A good resource to see what you are exposing to the outside world is here:
https://www.grc.com/x/ne.dll?bh0bkyd2

You should check it on a regular basis.

hannso · 10-11-2023, 02:40 AM

Thanks everyone!!
Your replies really helped me!
I will also need some time to digest your urls and tricks in detail. Looking forward to learn more on this.

Thanks again for your great support!

Peter_APIIT · 02-08-2024, 01:40 AM

Try disabled xineted service