Before wiping the server and starting over, thought I see if there was more I can do to find out about how this happened, get feedback on how to prevent.

Just happened to catch this:

sudo lsof -i | grep -i established
results:
sshd 2410 root 3u IPv4 14219 0t0 TCP 192.168.9.200:ssh->192.168.9.25:50393 (ESTABLISHED)
sshd 2412 root 3u IPv4 14235 0t0 TCP 192.168.9.200:ssh->116.10.191.234:6761 (ESTABLISHED)
sshd 2413 sshd 3u IPv4 14235 0t0 TCP 192.168.9.200:ssh->116.10.191.234:6761 (ESTABLISHED)
sshd 2573 db 3u IPv4 14219 0t0 TCP 192.168.9.200:ssh->192.168.9.25:50393 (ESTABLISHED)

192.168.9.25 is my remote PC IP.
116.10.191.234 originates in China.
Checked again a few minutes later and the connection for 116.10.191.234 was gone.

Ref: https://www.cert-bund.de/ebury-faq

116.10.191.234:6761 (ESTABLISHED) observed shortly (within an hour) after installing OpenSSH-Server. Auth.log shows:

db@emailserver:~$ cat /var/log/auth.log | grep -i accepted
May 16 19:38:17 emailserver sshd[9819]: Accepted password for db from 192.168.9.25 port 49756 ssh2
May 16 20:14:30 emailserver sshd[834]: Accepted password for db from 192.168.9.25 port 56239 ssh2
May 16 20:45:18 emailserver sshd[2410]: Accepted password for db from 192.168.9.25 port 50393 ssh2

db@emailserver:~$ cat /var/log/auth.log | grep -i 116.10.191.234
May 16 20:45:35 emailserver sshd[2412]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.10.191.234 user=root
May 16 20:45:37 emailserver sshd[2412]: Failed password for root from 116.10.191.234 port 6761 ssh2
May 16 20:45:40 emailserver sshd[2412]: Failed password for root from 116.10.191.234 port 6761 ssh2
May 16 20:45:42 emailserver sshd[2412]: Failed password for root from 116.10.191.234 port 6761 ssh2
May 16 20:45:49 emailserver sshd[2412]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.10.191.234 user=root
May 16 20:46:04 emailserver sshd[2675]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.10.191.234 user=root
May 16 20:46:06 emailserver sshd[2675]: Failed password for root from 116.10.191.234 port 15600 ssh2
May 16 20:46:13 emailserver sshd[2675]: PAM 3 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.10.191.234 user=root

Per reference article:
db@emailserver:~$ ssh -G 2>&1 | grep -e illegal -e unknown > /dev/null && echo "System clean" || echo "System infected"
System clean

However, per article, file size suggest Ebury is culprit:
db@emailserver:~$ find /lib* -type f -name libkeyutils.so* -exec ls -la {} \;
-rw-r--r-- 1 root root 13672 Oct 17 2011 /lib/i386-linux-gnu/libkeyutils.so.1.4

Additional notes: Server created 20APR14, connected to internet 24/7 for this entire time. No auth.log entries for failed connection attempts prior to installing openssh. Strong root password used.

The server is currently disconnected. Before wiping it clean and starting over, thought I would first check if it or I could be helpful to Linux community. Just need direction, any suggestions? Or, suggestions on how to prevent this after I start over? Repeat of 2002 OpenSSH trojan? I am fairly new to Linux, so in all likelyhood I just did some noob mistake.

Thanks for your support!

Indeed user "sshd" itself (as in 'getent passwd sshd') shouldn't be making such connections.



You forgot to include output of Next to that: verify your OpenSSH server and client and libkeyutils package.
Also note RKH version 1.4.2 includes 3 ClamAV signatures that may be of help. If you already have 'clamscan' installed (shouldn't install any software on presumed compromised machines) then you can extract the signatures, cd into the rkhunter/files/signatures directory (or supply full path to the "--database=" switch) and run
Next to properly hardening a server please adhere to SSH best practices:
- use sane SSH configuration values (no version 1 protocol, etc, etc),
- do not allow root logins over any network ever,
- only allow unprivileged user logins and using pubkey auth only,
- do deploy measures like fail2ban or equivalent.

Nothing showing for
ipcs -m

------ Shared Memory Segments --------
key shmid owner perms bytes nattch status


Since it appears there were no break-in attempts prior to installing OpenSSH, and then an immediate successful break-in shortly after, I suspect the Openssh install using apt-get to be the culprit. How does one check the source of the files being installed with apt-get are coming from a legitimate source, and even that they are trojan free?

Those might be just short-lived connections where someone was unsuccessfully trying to log in as root. If you have an ssh server listening on port 22 you are going to see a lot of those.

Do I understand correctly, these entries for sudo lsof -i | grep -i established

sshd 2412 root 3u IPv4 14235 0t0 TCP 192.168.9.200:ssh->116.10.191.234:6761 (ESTABLISHED)
sshd 2413 sshd 3u IPv4 14235 0t0 TCP 192.168.9.200:ssh->116.10.191.234:6761 (ESTABLISHED)

could have been short lived connections for break-in attempts, and not an actual successful break-in?

Was the ipcs command run as root user?
Or did you run the command right after restarting the SSH daemon?


There's several ways and the default Debian(-like) way of using debsums unfortunately requires you to modify apt-related configuration files before the event. One way is to download a copy of the SSH daemon, client, dependencies and any other packages onto a known clean machine, generate a list of hashes (md5sum or md5deep) and run them with 'md5sum -c' on the "victim" system.


Did you check all logs? Have you verified the integrity of the whole system? If not then please be more thorough and don't be too quick drawing conclusions.

Oops, a red herring just landed in the boat. I simply don't accept this hypothesis.

I would agree with this seeing how my system shows sshd as the user for unsuccessful logins. Plus with your Auth log, it shows the China ip as trying to log in as root. So ignoring the unsuccessful notice there would mean that if there was access then the root account would be shown with a second ip address as a established connection with ssh. Also Iirc sshd as a user has no valid shell to run like a standalone ssh user (such as /bin/bash for root). You can check this by looking into / etc / passwd at the shell column. That being said definitely run ssh on another port/ harden the ssh config/ install fail2ban or similar.

Yes. The TCP handshake has to establish a connection before any login attempt can begin.

Agreed, the hypothesis was incorrect.
After you kind and insightful suggestions, I ran experiment, connect with SSH but not log in.
The result was an additional established connection, which went away when closing the not logged in SSH session.
So in fact, no proof of a successful intrusion is presented.
Thank you all for your support!

1 members found this post helpful.

But you do have evidence in logs of password guessing against your site. Consider disabling password logins and requiring keys to get rid of this.

"Whaddaya mean, consider?!"

When you go to any workplace, you do not see a guard saying, "What's the magic word?" No, you see a badge-reader. Everyone's got their own badge, uniquely issued to them.

Also, in many companies, you have to pass through a front-gate and a fence just to get into the complex at all. In computerland, that gate is called "VPN," and you should be using it to encircle everything. There are no "open ports," unless and until you - again, using your uniquely-issued identification badge, not a password (PSK) – can pass over the one drawbridge that crosses the outer moat. Only then are you privileged to use your SSH credentials to pass further.

It is called, "defense in depth." If users are able even to attempt an ssh-login into your system from afar, then you are already doing something very wrong.

Thanks everyone. Responses ranged from skeptical to ultra paranoid, and I learned a lot! Security is far more complex than I first imagined, and more interesting as well. Finding the right balance between paranoid, which can be difficult to manage, and insufficient, which can be difficult to recover from, is the challenge. For my server, and for its purpose, I am choosing the following:

iptables - firewall

Fail2Ban - set to permanently ban on three log in failure attempts. Did this by changing the default tries of 5 to 3 and ban timeout to 0 (infinite). I use a very strong password, so this is really overkill. Mostly it annoys me that failed log in attempts are clogging up my log files. I use sudo, so root and ssh users have log in disabled. There are hardly any other users that are attempted for break in attempts.

Obscurity - changed the ssh port from default to some other large number under the max of 65,535.

Tripwire - let's me know who gains access and when. If they weren't supposed to, I can revert to backup and change passwords.

md5 checksums - get a clean list of check sums after all software is installed, and before connecting to the internet and save to USB or remote location. This way I can periodically check the installed applications to verify their integrity in case crafty intruder managed to sneak in and root the server. Looks like rkhunter might have some functionality to automate this, so I'll look into that.

ssh - using private key with pass phrase and disabled password log in capability for all users.

I'm also considering one other suggestions posted on another site, use separate server to monitor network traffic. Since Tripwire could be compromised if the server is broken into, this would be another option for a more secure way to monitor connections. It however is crossing the line into ultra paranoid, and still, seems like an interesting thing to explore.

Also, an idea I'm toying with is creating a simple script that would monitor the 'who' output, and trigger a timer that when expired will disable all NICs. The script would have to be disabled shortly after logging in to prevent the server from being cut off from the outside world. Crafty intruder would have to have first hand knowledge of such a script in order to defeat it. Again, major overkill, but simple to implement, and easy to maintain.

Finally, if you made it this far, then you will certainly enjoy the following article, which chronicles the adventures of a security expert who sets up servers with the goal of baiting would be crafty intruders to analyze what they do. In this case it is a person or organization that utilizes servers to effect DOS attacks:

http://draios.com/fishing-for-hackers/