[SOLVED] Does this indicate I was broken into?

haertig · 01-18-2017, 10:46 PM

I just happened to run a netstat command, and puckered up when I saw this:

Code:

# netstat -anp | grep EST
tcp        0      0 10.192.0.2:42036        74.125.129.109:993      ESTABLISHED 31966/thunderbird
tcp        0      0 10.192.0.2:22           95.49.118.165:52762     ESTABLISHED 551/sshd: [accepted
tcp        0      0 10.192.0.2:36772        216.58.217.46:443       ESTABLISHED 32441/chrome    
tcp        0      0 10.192.0.2:42056        74.125.129.109:993      ESTABLISHED 31966/thunderbird
#

Nobody ssh's into my system except me, it's a single user system, so I was very worried. I followed up with this:

Code:

# traceroute 95.49.118.165
traceroute to 95.49.118.165 (95.49.118.165), 30 hops max, 60 byte packets
 1  router (10.192.0.1)  0.233 ms  0.369 ms  0.445 ms
 2  96.120.13.141 (96.120.13.141)  9.491 ms  15.772 ms  14.812 ms
 3  ae-212-sur03.arvada.co.denver.comcast.net (162.151.38.181)  16.770 ms  16.810 ms  16.866 ms
 4  ae-21-ar01.denver.co.denver.comcast.net (68.86.179.57)  17.735 ms  17.738 ms  17.787 ms
 5  be-33652-cr01.1601milehigh.co.ibone.comcast.net (68.86.92.121)  17.884 ms * *
 6  be-11719-cr02.denver.co.ibone.comcast.net (68.86.86.77)  19.707 ms  16.391 ms  15.629 ms
 7  be-10517-cr02.350ecermak.il.ibone.comcast.net (68.86.85.169)  38.678 ms  33.852 ms  34.886 ms
 8  be-10563-pe01.350ecermak.il.ibone.comcast.net (68.86.82.158)  39.538 ms  39.571 ms  38.556 ms
 9  bundle-ether6.chitr1.Chicago.opentransit.net (193.251.248.223)  41.348 ms  42.275 ms  42.307 ms
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  war-bg2.tpnet.pl (80.50.156.50)  167.512 ms  168.508 ms  168.423 ms
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
#

Not sure what .pl is, probably Poland. Getting more worried now. Followed up next with:

Code:

# pkill sshd

Went to ipaddress.com and looked up that IP. Yep ... Poland.

Next:

Code:

# last

wtmp begins Sun Jan  1 07:35:02 2017
#

Code:

# lastlog
Username         Port     From             Latest
root                                       **Never logged in**
daemon                                     **Never logged in**
bin                                        **Never logged in**
sys                                        **Never logged in**
sync                                       **Never logged in**
games                                      **Never logged in**
man                                        **Never logged in**
lp                                         **Never logged in**
mail                                       **Never logged in**
news                                       **Never logged in**
uucp                                       **Never logged in**
proxy                                      **Never logged in**
www-data                                   **Never logged in**
backup                                     **Never logged in**
list                                       **Never logged in**
irc                                        **Never logged in**
gnats                                      **Never logged in**
nobody                                     **Never logged in**
systemd-timesync                           **Never logged in**
systemd-network                            **Never logged in**
systemd-resolve                            **Never logged in**
systemd-bus-proxy                           **Never logged in**
syslog                                     **Never logged in**
_apt                                       **Never logged in**
messagebus                                 **Never logged in**
uuidd                                      **Never logged in**
ntp                                        **Never logged in**
avahi-autoipd                              **Never logged in**
avahi                                      **Never logged in**
colord                                     **Never logged in**
dnsmasq                                    **Never logged in**
speech-dispatcher                           **Never logged in**
hplip                                      **Never logged in**
kernoops                                   **Never logged in**
pulse                                      **Never logged in**
nm-openvpn                                 **Never logged in**
rtkit                                      **Never logged in**
saned                                      **Never logged in**
usbmux                                     **Never logged in**
mdm                                        **Never logged in**
david            pts/1    135.9.203.90     Tue Nov 29 15:18:07 -0700 2016
plex                                       **Never logged in**
sshd                                       **Never logged in**
nx                                         **Never logged in**
#

Running the "history" command for both 'root' and 'david' showed nothing out of the ordinary. Everything I found there was expected.

I thought I had my sshd locked down pretty good. I thought I had it set up to only allow pubkey authentication, so no password guessing would be allowed (and I have a strong password anyway).

Does this sshd configuration really lock things down to only allow pubkey authentication as I thought?

Code:

# cat /etc/ssh/sshd_config
# Package generated configuration file
# See the sshd_config(5) manpage for details

# What ports, IPs and protocols we listen for
Port 22
# Use these options to restrict which interfaces/protocols sshd will bind to
#ListenAddress ::
#ListenAddress 0.0.0.0
Protocol 2
# HostKeys for protocol version 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
#Privilege Separation is turned on for security
UsePrivilegeSeparation yes

# Lifetime and size of ephemeral version 1 server key
KeyRegenerationInterval 3600
ServerKeyBits 1024

# Logging
SyslogFacility AUTH
LogLevel INFO

# Authentication:
#LoginGraceTime 120
LoginGraceTime 30
#PermitRootLogin prohibit-password
PermitRootLogin no
StrictModes yes

#RSAAuthentication yes
RSAAuthentication no
PubkeyAuthentication yes
# Added 8/13/2016 to allow logins using older DSS keys
PubkeyAcceptedKeyTypes=+ssh-dss
#AuthorizedKeysFile	%h/.ssh/authorized_keys

AllowUsers david nx

# Don't read the user's ~/.rhosts and ~/.shosts files
IgnoreRhosts yes
# For this to work you will also need host keys in /etc/ssh_known_hosts
RhostsRSAAuthentication no
# similar for protocol version 2
HostbasedAuthentication no
# Uncomment if you don't trust ~/.ssh/known_hosts for RhostsRSAAuthentication
#IgnoreUserKnownHosts yes

# To enable empty passwords, change to yes (NOT RECOMMENDED)
PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Change to no to disable tunnelled clear text passwords
#PasswordAuthentication yes
PasswordAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosGetAFSToken no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes

X11Forwarding yes
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
#UseLogin no

#MaxStartups 10:30:60
#Banner /etc/issue.net

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

Subsystem sftp /usr/lib/openssh/sftp-server

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
#UsePAM yes
UsePAM no
#

I'm stumped as to how someone might have gotten in. pubkeys are pretty secure. Does that ESTABLISHED connection in netstat output really indicate that someone might have gotten in, or just that they were connected and TRYING to get in?

I see a bunch of unexpected entries in my auth.log. This is just a sample of them - there were many more, from different IP's. The time period for these entries is pretty short. Are there that many people knocking on my front door?

Code:

Jan 18 20:18:43 Davids-Linux-Desktop sshd[31631]: Did not receive identification string from 79.101.95.120
Jan 18 20:18:51 Davids-Linux-Desktop sshd[31636]: Did not receive identification string from 190.62.104.199
Jan 18 20:19:36 Davids-Linux-Desktop sshd[31663]: Did not receive identification string from 189.218.126.97
Jan 18 20:22:17 Davids-Linux-Desktop sshd[31834]: Did not receive identification string from 178.54.221.26
Jan 18 20:22:27 Davids-Linux-Desktop sshd[31835]: Did not receive identification string from 114.34.187.33
Jan 18 20:24:05 Davids-Linux-Desktop sshd[31894]: Did not receive identification string from 118.69.32.184

Thanks in advance for any advice that is given.

TheEzekielProject · 01-18-2017, 11:54 PM

Quote:

# cat /etc/ssh/sshd_config
# Package generated configuration file
# See the sshd_config(5) manpage for details

# What ports, IPs and protocols we listen for
Port 22
# Use these options to restrict which interfaces/protocols sshd will bind to
#ListenAddress ::
#ListenAddress 0.0.0.0
Protocol 2
# HostKeys for protocol version 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
#Privilege Separation is turned on for security
UsePrivilegeSeparation yes

# Lifetime and size of ephemeral version 1 server key
KeyRegenerationInterval 3600
ServerKeyBits 1024

# Logging
SyslogFacility AUTH
LogLevel INFO

# Authentication:
#LoginGraceTime 120
LoginGraceTime 30
#PermitRootLogin prohibit-password
PermitRootLogin no
StrictModes yes

#RSAAuthentication yes
RSAAuthentication no
PubkeyAuthentication yes
# Added 8/13/2016 to allow logins using older DSS keys
PubkeyAcceptedKeyTypes=+ssh-dss
#AuthorizedKeysFile %h/.ssh/authorized_keys

AllowUsers david nx

# Don't read the user's ~/.rhosts and ~/.shosts files
IgnoreRhosts yes
# For this to work you will also need host keys in /etc/ssh_known_hosts
RhostsRSAAuthentication no
# similar for protocol version 2
HostbasedAuthentication no
# Uncomment if you don't trust ~/.ssh/known_hosts for RhostsRSAAuthentication
#IgnoreUserKnownHosts yes

# To enable empty passwords, change to yes (NOT RECOMMENDED)
PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Change to no to disable tunnelled clear text passwords
#PasswordAuthentication yes
PasswordAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosGetAFSToken no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes

X11Forwarding yes
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
#UseLogin no

#MaxStartups 10:30:60
#Banner /etc/issue.net

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

Subsystem sftp /usr/lib/openssh/sftp-server

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication. Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
#UsePAM yes
UsePAM no
#

Please see bolded!
Highly recommend changing Port 22 to something else!
Highly recommend key size of 2048! (Or 4096!)
Rather than making another line that says yes rather than no (or vice versa) just change the yes or no!

I also recommend fail2ban!

is user "nx" you?

haertig · 01-19-2017, 12:56 AM

Thanks for the reply.

When I change a default setting in sshd_config, I leave the default setting there, commented out, so that I can quickly see what the default was and what I changed it's value to. Having the original default line still in the file, but commented out, doesn't hurt anything. It doesn't affect security either.

The port 22 you see is accessible from my LAN. To get there from the WAN, you must go through my router which forwards a different WAN port to port 22 on this LAN computer. So effectively, I have already "changed the port". But changing the port really does nothing to add security. It would be a very stupid hacker who thought that SSH could only run on port 22. Changing from port 22 to something else might delay a hackers port scan by about 18 milliseconds, but not much else.

ServerKeyBits is set to 1024 because that was the default. I didn't change that because ServerKeyBits is only used with SSH "Protocol 1". You can see in my config that I only allow "Protocol 2", so ServerKeyBits is not used, and ignored.

I am vaguely familiar with fail2ban. I believe it works on similar principles to a script I wrote and posted here on these forums over a decade ago. Named "banssh". For all I know, fail2ban might have even been based on my script or Bob Toxen's original concept that I based my script on (Bob was the instructor in a Linux security course that I took many years ago - fantastic course, Bob really knows his stuff!)

Start with this post: http://www.linuxquestions.org/questi...6/#post2290227 and then read on down two or three posts after that to see the script.

I stopped using my banssh script a few years ago because I stopped allowing anything from the WAN into my LAN except VPN. So if I wanted to SSH to one of my internal computers, I would first have to VPN into my LAN (my router runs OpenVPN) and then from there, SSH to the target computer. With that setup, I no longer needed banssh. But recently, I opened up port 22 on one LAN computer to the WAN via router port forwarding. And to be honest, I just plain forgot to turn banssh back on for this newly exposed computer. That was my error. Thanks for reminding me.

User "nx" is used by the NoMachine NX server (GUI remote access, similar to VNC, but much faster). While installed, that server has never been turned on. I use the client for NX to connect out to remote computers I manage. But nobody connects incoming to my computer using NX. The server is installed, because the client shares some components with it. But the server is not running. You're seeing one of the footprints of the server being installed. The reference to the "nx" userid.

At one time in the past, I was pretty well versed in setting up SSH securely. That was a long time ago, and in recent years I've pretty much just carried over my older implementations. I've lost my edge on secure SSH setup (hence my questions in this thread!) When I saw that ESTABLISHED connection to port 22, that grabbed my attention real fast. I was thinking to myself, "Hey, I thought I already had all that nonsense covered!" Evidently not. Or maybe I'm reading too much into that ESTABLISHED connection. That's why I'm asking questions in this thread. To re-learn what I used to know, get up to speed on more recent updates and security techniques, etc.

Thanks again for the reply. I appreciate it. Especially when this thread started out with a detailed and long winded post. Those are difficult to wade through, I know. And here I am AGAIN, with ANOTHER long winded post!

TheEzekielProject · 01-19-2017, 02:12 AM

No worries! I was not sure if that meant anyone had connected or not, I was just making some general security change suggestions I often found when I first started learning about all that. I agree changing the default port isn't necessarily going to slow down a targeted attack by any measure, but I'd argue that one would see less login attempts in general. After all, it'd be lot faster to

Code:

nmap -p 22

a given range of ip addresses than to

Code:

nmap -p 1-65535

But, as you said, you have already taken care of this!

I will note for what it's worth that running the same netstat command on my system shows the username of the connected user after "ESTABLISHED 551/sshd:" rather than "[accepted", or, if there is no one connected it does not appear at all

descendant_command · 01-19-2017, 02:13 AM

netstat is just showing the ESTABLISHED TCP connection - your auth.log shows the SSH negotiation failing as presumably the bot on the other end is trying to supply a user/pass or simply waiting for a prompt until it times out.

TheEzekielProject · 01-19-2017, 02:50 AM

After playing around with logins a bit, I was able to replicate the output ESTABLISHED ***/sshd: [accepted.

It appears this indeed was someone connected but NOT authenticated.

I did this just by ssh'ing to my ipaddress from let's call it computer B, at which point I received the prompt showing the ECDSA key fingerprint and asking if I wanted to accept. Before answering, on computer A I ran

Code:

netstat -anp | grep EST
tcp        0      0 192.168.1.228:22     192.168.1.12:60216      ESTABLISHED 30670/sshd: [accept

Note that after answering whether or not to continue connecting I no longer see sshd from the netstat command when pubkey authentication is enabled and any further connection attempts fail.

When pubkey authentication is NOT enabled, and after answering whether to continue connecting, the username appears in place of "[accept"

Habitual · 01-19-2017, 09:38 AM

nx = nomachine?

haertig · 01-19-2017, 09:43 AM

Thanks for the help guys!

I guess I overreacted when I saw that ESTABLISHED connection on port 22. I don't run netstat routinely ... only if I have some reason to look at what's going on there. Yesterday, I was looking (for something else), saw the port 22 connection, and puckered up a little. I guess I've never stumbled onto an active (but ultimately repelled) SSH hack attempt. Had my brain been working correctly, the analysis should have been obvious. Of course you have to connect before you can authenticate!

I guess floundering is good for you every now and then. It gets those stagnant brain cells woken up. Makes you question whether you really know what you're doing like you thought you did. It's good to question every now and then, to keep yourself sharp.

haertig · 01-19-2017, 09:47 AM

Yes, nx is NoMachine. I use their software to manage other friends/family computers. I use an older version because it is all peer-to-peer. I believe the newer versions of NoMachine connect you through their central servers as middlemen. I prefer not to have a middleman computer in the remote connection mix. The older versions of NoMachine use the nx userid in their implementation. I don't know if the newer versions use that. NoMachine NX is much faster than VNC.

haertig · 01-19-2017, 03:37 PM

OK, obviously earlier I just got prematurely freaked out by a run of the mill and harmless port scan.

Here's some example netstat output (the netstat command was run once per second) illustrating a scan in progress on the computer I'm typing on right now. You can see that every few seconds the process id of sshd is changing, as is the port on the remote end, indicating they're knocking on the door, but not getting in.

Code:

# netstat -anp | grep EST | grep :22
tcp        0      0 10.192.0.2:22           146.255.230.86:51327    ESTABLISHED 14660/sshd: [accept
# netstat -anp | grep EST | grep :22
tcp        0      0 10.192.0.2:22           146.255.230.86:51327    ESTABLISHED 14660/sshd: [accept
# netstat -anp | grep EST | grep :22
tcp        0      0 10.192.0.2:22           146.255.230.86:51327    ESTABLISHED 14660/sshd: [accept
# netstat -anp | grep EST | grep :22
tcp        0      0 10.192.0.2:22           146.255.230.86:51327    ESTABLISHED 14660/sshd: [accept
# netstat -anp | grep EST | grep :22
tcp        0      0 10.192.0.2:22           146.255.230.86:51327    ESTABLISHED 14660/sshd: [accept
# netstat -anp | grep EST | grep :22
tcp        0      0 10.192.0.2:22           146.255.230.86:52232    ESTABLISHED 14700/sshd: [accept
# netstat -anp | grep EST | grep :22
tcp        0      0 10.192.0.2:22           146.255.230.86:52232    ESTABLISHED 14700/sshd: [accept
# netstat -anp | grep EST | grep :22
tcp        0      0 10.192.0.2:22           146.255.230.86:52232    ESTABLISHED 14700/sshd: [accept
# netstat -anp | grep EST | grep :22
tcp        0      0 10.192.0.2:22           146.255.230.86:52232    ESTABLISHED 14700/sshd: [accept
# netstat -anp | grep EST | grep :22
tcp        0      0 10.192.0.2:22           146.255.230.86:52232    ESTABLISHED 14700/sshd: [accept
# netstat -anp | grep EST | grep :22
tcp        0      0 10.192.0.2:22           146.255.230.86:52388    ESTABLISHED 14731/sshd: [accept
# netstat -anp | grep EST | grep :22
tcp        0      0 10.192.0.2:22           146.255.230.86:52388    ESTABLISHED 14731/sshd: [accept
# netstat -anp | grep EST | grep :22
tcp        0      0 10.192.0.2:22           146.255.230.86:52388    ESTABLISHED 14731/sshd: [accept
# netstat -anp | grep EST | grep :22
tcp        0      0 10.192.0.2:22           146.255.230.86:52388    ESTABLISHED 14731/sshd: [accept
# netstat -anp | grep EST | grep :22
tcp        0      0 10.192.0.2:22           146.255.230.86:52388    ESTABLISHED 14731/sshd: [accept
# netstat -anp | grep EST | grep :22
tcp        0      0 10.192.0.2:22           146.255.230.86:52388    ESTABLISHED 14731/sshd: [accept
# netstat -anp | grep EST | grep :22
tcp        0      0 10.192.0.2:22           146.255.230.86:52388    ESTABLISHED 14731/sshd: [accept
# netstat -anp | grep EST | grep :22
tcp        0      0 10.192.0.2:22           146.255.230.86:52388    ESTABLISHED 14731/sshd: [accept
# netstat -anp | grep EST | grep :22
tcp        0      0 10.192.0.2:22           146.255.230.86:52388    ESTABLISHED 14731/sshd: [accept
# netstat -anp | grep EST | grep :22
tcp        0      0 10.192.0.2:22           146.255.230.86:52583    ESTABLISHED 14783/sshd: [accept
# netstat -anp | grep EST | grep :22
tcp        0      0 10.192.0.2:22           146.255.230.86:52583    ESTABLISHED 14783/sshd: [accept
# netstat -anp | grep EST | grep :22
tcp        0      0 10.192.0.2:22           146.255.230.86:52583    ESTABLISHED 14783/sshd: [accept
# netstat -anp | grep EST | grep :22
tcp        0      0 10.192.0.2:22           146.255.230.86:52583    ESTABLISHED 14783/sshd: [accept
#

Code:

# nslookup 146.255.230.86
Server:		127.0.1.1
Address:	127.0.1.1#53

Non-authoritative answer:
86.230.255.146.in-addr.arpa	name = 146-255-230-86.customers.silknet.com.

Authoritative answers can be found from:

#

Code:

# whois 146.255.230.86

...

OrgName:        RIPE Network Coordination Centre
OrgId:          RIPE
Address:        P.O. Box 10096
City:           Amsterdam

...

#

Coffee!!! · 01-20-2017, 09:39 AM

I recommend creating a script to create hashes of /etc and /usr after every set of patches so you can compare the checksums. This will allow you to verify if anything has been altered on your system. If you want to go the package route, you can install AIDE and create a cronjob to notify you of any changes. Very helpful in the event you do get hacked, you can see what binaries or configurations have been changed.