LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices

Reply
 
Search this Thread
Old 05-17-2014, 01:49 AM   #1
vRanger
LQ Newbie
 
Registered: May 2014
Posts: 7

Rep: Reputation: Disabled
Suspect sever break-in with user 'sshd' on Ubuntu 12.04, Ebury?


Before wiping the server and starting over, thought I see if there was more I can do to find out about how this happened, get feedback on how to prevent.

Just happened to catch this:

sudo lsof -i | grep -i established
results:
sshd 2410 root 3u IPv4 14219 0t0 TCP 192.168.9.200:ssh->192.168.9.25:50393 (ESTABLISHED)
sshd 2412 root 3u IPv4 14235 0t0 TCP 192.168.9.200:ssh->116.10.191.234:6761 (ESTABLISHED)
sshd 2413 sshd 3u IPv4 14235 0t0 TCP 192.168.9.200:ssh->116.10.191.234:6761 (ESTABLISHED)
sshd 2573 db 3u IPv4 14219 0t0 TCP 192.168.9.200:ssh->192.168.9.25:50393 (ESTABLISHED)

192.168.9.25 is my remote PC IP.
116.10.191.234 originates in China.
Checked again a few minutes later and the connection for 116.10.191.234 was gone.

Ref: https://www.cert-bund.de/ebury-faq

116.10.191.234:6761 (ESTABLISHED) observed shortly (within an hour) after installing OpenSSH-Server. Auth.log shows:

db@emailserver:~$ cat /var/log/auth.log | grep -i accepted
May 16 19:38:17 emailserver sshd[9819]: Accepted password for db from 192.168.9.25 port 49756 ssh2
May 16 20:14:30 emailserver sshd[834]: Accepted password for db from 192.168.9.25 port 56239 ssh2
May 16 20:45:18 emailserver sshd[2410]: Accepted password for db from 192.168.9.25 port 50393 ssh2

db@emailserver:~$ cat /var/log/auth.log | grep -i 116.10.191.234
May 16 20:45:35 emailserver sshd[2412]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.10.191.234 user=root
May 16 20:45:37 emailserver sshd[2412]: Failed password for root from 116.10.191.234 port 6761 ssh2
May 16 20:45:40 emailserver sshd[2412]: Failed password for root from 116.10.191.234 port 6761 ssh2
May 16 20:45:42 emailserver sshd[2412]: Failed password for root from 116.10.191.234 port 6761 ssh2
May 16 20:45:49 emailserver sshd[2412]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.10.191.234 user=root
May 16 20:46:04 emailserver sshd[2675]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.10.191.234 user=root
May 16 20:46:06 emailserver sshd[2675]: Failed password for root from 116.10.191.234 port 15600 ssh2
May 16 20:46:13 emailserver sshd[2675]: PAM 3 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.10.191.234 user=root

Per reference article:
db@emailserver:~$ ssh -G 2>&1 | grep -e illegal -e unknown > /dev/null && echo "System clean" || echo "System infected"
System clean

However, per article, file size suggest Ebury is culprit:
db@emailserver:~$ find /lib* -type f -name libkeyutils.so* -exec ls -la {} \;
-rw-r--r-- 1 root root 13672 Oct 17 2011 /lib/i386-linux-gnu/libkeyutils.so.1.4

Additional notes: Server created 20APR14, connected to internet 24/7 for this entire time. No auth.log entries for failed connection attempts prior to installing openssh. Strong root password used.

The server is currently disconnected. Before wiping it clean and starting over, thought I would first check if it or I could be helpful to Linux community. Just need direction, any suggestions? Or, suggestions on how to prevent this after I start over? Repeat of 2002 OpenSSH trojan? I am fairly new to Linux, so in all likelyhood I just did some noob mistake.

Thanks for your support!

Last edited by vRanger; 05-17-2014 at 09:17 AM.
 
Old 05-17-2014, 03:27 AM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,447
Blog Entries: 54

Rep: Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890
Quote:
Originally Posted by vranger@genzyme.linkpc.n View Post
Code:
sudo lsof -i | grep -i established 
results:
sshd      2412        root    3u  IPv4  14235      0t0  TCP 192.168.9.200:ssh->116.10.191.234:6761 (ESTABLISHED)
sshd      2413        sshd    3u  IPv4  14235      0t0  TCP 192.168.9.200:ssh->116.10.191.234:6761 (ESTABLISHED)
Indeed user "sshd" itself (as in 'getent passwd sshd') shouldn't be making such connections.


Quote:
Originally Posted by vranger@genzyme.linkpc.n View Post
Code:
db@emailserver:~$ cat /var/log/auth.log | grep -i 116.10.191.234
May 16 20:45:35 emailserver sshd[2412]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.10.191.234  user=root
May 16 20:45:37 emailserver sshd[2412]: Failed password for root from 116.10.191.234 port 6761 ssh2
May 16 20:45:40 emailserver sshd[2412]: Failed password for root from 116.10.191.234 port 6761 ssh2
May 16 20:45:42 emailserver sshd[2412]: Failed password for root from 116.10.191.234 port 6761 ssh2
May 16 20:45:49 emailserver sshd[2412]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.10.191.234  user=root
May 16 20:46:04 emailserver sshd[2675]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.10.191.234  user=root
May 16 20:46:06 emailserver sshd[2675]: Failed password for root from 116.10.191.234 port 15600 ssh2
May 16 20:46:13 emailserver sshd[2675]: PAM 3 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.10.191.234  user=root

Quote:
Originally Posted by vranger@genzyme.linkpc.n View Post
However, per article, file size suggest Ebury is culprit:
Code:
db@emailserver:~$ find /lib* -type f -name libkeyutils.so* -exec ls -la {} \;
-rw-r--r-- 1 root root 13672 Oct 17  2011 /lib/i386-linux-gnu/libkeyutils.so.1.4
You forgot to include output of
Code:
ipcs -m
Next to that: verify your OpenSSH server and client and libkeyutils package.
Also note RKH version 1.4.2 includes 3 ClamAV signatures that may be of help. If you already have 'clamscan' installed (shouldn't install any software on presumed compromised machines) then you can extract the signatures, cd into the rkhunter/files/signatures directory (or supply full path to the "--database=" switch) and run
Code:
clamscan --database=RKH_sshd.ldb --database=RKH_libkeyutils.ldb --database=RKH_libkeyutils1.ldb -i /usr/sbin/sshd /usr/lib/libkeyutils.so*

Quote:
Originally Posted by vranger@genzyme.linkpc.n View Post
Additional notes: Server created 20APR14, connected to internet 24/7 for this entire time. No auth.log entries for failed connection attempts prior to installing openssh. Strong root password used.
Next to properly hardening a server please adhere to SSH best practices:
- use sane SSH configuration values (no version 1 protocol, etc, etc),
- do not allow root logins over any network ever,
- only allow unprivileged user logins and using pubkey auth only,
- do deploy measures like fail2ban or equivalent.
 
Old 05-19-2014, 10:00 AM   #3
vRanger
LQ Newbie
 
Registered: May 2014
Posts: 7

Original Poster
Rep: Reputation: Disabled
Nothing showing for
ipcs -m

------ Shared Memory Segments --------
key shmid owner perms bytes nattch status


Since it appears there were no break-in attempts prior to installing OpenSSH, and then an immediate successful break-in shortly after, I suspect the Openssh install using apt-get to be the culprit. How does one check the source of the files being installed with apt-get are coming from a legitimate source, and even that they are trojan free?
 
Old 05-19-2014, 01:18 PM   #4
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 1,484

Rep: Reputation: 611Reputation: 611Reputation: 611Reputation: 611Reputation: 611Reputation: 611
Those might be just short-lived connections where someone was unsuccessfully trying to log in as root. If you have an ssh server listening on port 22 you are going to see a lot of those.
 
Old 05-19-2014, 01:26 PM   #5
vRanger
LQ Newbie
 
Registered: May 2014
Posts: 7

Original Poster
Rep: Reputation: Disabled
Do I understand correctly, these entries for sudo lsof -i | grep -i established

sshd 2412 root 3u IPv4 14235 0t0 TCP 192.168.9.200:ssh->116.10.191.234:6761 (ESTABLISHED)
sshd 2413 sshd 3u IPv4 14235 0t0 TCP 192.168.9.200:ssh->116.10.191.234:6761 (ESTABLISHED)

could have been short lived connections for break-in attempts, and not an actual successful break-in?
 
Old 05-20-2014, 01:38 AM   #6
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,447
Blog Entries: 54

Rep: Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890Reputation: 2890
Quote:
Originally Posted by vRanger View Post
Nothing showing for
ipcs -m

------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
Was the ipcs command run as root user?
Or did you run the command right after restarting the SSH daemon?


Quote:
Originally Posted by vRanger View Post
How does one check the source of the files being installed with apt-get are coming from a legitimate source, and even that they are trojan free?
There's several ways and the default Debian(-like) way of using debsums unfortunately requires you to modify apt-related configuration files before the event. One way is to download a copy of the SSH daemon, client, dependencies and any other packages onto a known clean machine, generate a list of hashes (md5sum or md5deep) and run them with 'md5sum -c' on the "victim" system.


Quote:
Originally Posted by vRanger View Post
Since it appears there were no break-in attempts prior to installing OpenSSH, and then an immediate successful break-in shortly after, I suspect the Openssh install using apt-get to be the culprit.
Did you check all logs? Have you verified the integrity of the whole system? If not then please be more thorough and don't be too quick drawing conclusions.
 
Old 05-20-2014, 06:20 AM   #7
sundialsvcs
Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 5,397

Rep: Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114
Quote:
Since it appears there were no break-in attempts prior to installing OpenSSH, and then an immediate successful break-in shortly after, I suspect the Openssh install using apt-get to be the culprit.
Oops, a red herring just landed in the boat. I simply don't accept this hypothesis.
 
Old 05-20-2014, 06:39 AM   #8
antiX-Dave
LQ Newbie
 
Registered: Dec 2013
Distribution: antiX
Posts: 13

Rep: Reputation: Disabled
Quote:
Originally Posted by vRanger View Post
Do I understand correctly, these entries for sudo lsof -i | grep -i established

sshd 2412 root 3u IPv4 14235 0t0 TCP 192.168.9.200:ssh->116.10.191.234:6761 (ESTABLISHED)
sshd 2413 sshd 3u IPv4 14235 0t0 TCP 192.168.9.200:ssh->116.10.191.234:6761 (ESTABLISHED)

could have been short lived connections for break-in attempts, and not an actual successful break-in?
I would agree with this seeing how my system shows sshd as the user for unsuccessful logins. Plus with your Auth log, it shows the China ip as trying to log in as root. So ignoring the unsuccessful notice there would mean that if there was access then the root account would be shown with a second ip address as a established connection with ssh. Also Iirc sshd as a user has no valid shell to run like a standalone ssh user (such as /bin/bash for root). You can check this by looking into / etc / passwd at the shell column. That being said definitely run ssh on another port/ harden the ssh config/ install fail2ban or similar.
 
Old 05-20-2014, 08:03 AM   #9
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 1,484

Rep: Reputation: 611Reputation: 611Reputation: 611Reputation: 611Reputation: 611Reputation: 611
Quote:
Originally Posted by vRanger View Post
Do I understand correctly, these entries for sudo lsof -i | grep -i established

sshd 2412 root 3u IPv4 14235 0t0 TCP 192.168.9.200:ssh->116.10.191.234:6761 (ESTABLISHED)
sshd 2413 sshd 3u IPv4 14235 0t0 TCP 192.168.9.200:ssh->116.10.191.234:6761 (ESTABLISHED)

could have been short lived connections for break-in attempts, and not an actual successful break-in?
Yes. The TCP handshake has to establish a connection before any login attempt can begin.
 
Old 05-20-2014, 12:33 PM   #10
vRanger
LQ Newbie
 
Registered: May 2014
Posts: 7

Original Poster
Rep: Reputation: Disabled
Agreed, the hypothesis was incorrect.
After you kind and insightful suggestions, I ran experiment, connect with SSH but not log in.
The result was an additional established connection, which went away when closing the not logged in SSH session.
So in fact, no proof of a successful intrusion is presented.
Thank you all for your support!
 
1 members found this post helpful.
Old 06-02-2014, 10:27 AM   #11
linosaurusroot
Member
 
Registered: Oct 2012
Distribution: OpenSuSE,RHEL,Fedora,OpenBSD
Posts: 801
Blog Entries: 2

Rep: Reputation: 203Reputation: 203Reputation: 203
But you do have evidence in logs of password guessing against your site. Consider disabling password logins and requiring keys to get rid of this.
 
Old 06-06-2014, 07:36 AM   #12
sundialsvcs
Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 5,397

Rep: Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114Reputation: 1114
Quote:
Originally Posted by linosaurusroot View Post
But you do have evidence in logs of password guessing against your site. Consider disabling password logins and requiring keys to get rid of this.
"Whaddaya mean, consider?!"

When you go to any workplace, you do not see a guard saying, "What's the magic word?" No, you see a badge-reader. Everyone's got their own badge, uniquely issued to them.

Also, in many companies, you have to pass through a front-gate and a fence just to get into the complex at all. In computerland, that gate is called "VPN," and you should be using it to encircle everything. There are no "open ports," unless and until you - again, using your uniquely-issued identification badge, not a password (PSK) – can pass over the one drawbridge that crosses the outer moat. Only then are you privileged to use your SSH credentials to pass further.

It is called, "defense in depth." If users are able even to attempt an ssh-login into your system from afar, then you are already doing something very wrong.
 
Old 06-09-2014, 10:48 AM   #13
vRanger
LQ Newbie
 
Registered: May 2014
Posts: 7

Original Poster
Rep: Reputation: Disabled
Thanks everyone. Responses ranged from skeptical to ultra paranoid, and I learned a lot! Security is far more complex than I first imagined, and more interesting as well. Finding the right balance between paranoid, which can be difficult to manage, and insufficient, which can be difficult to recover from, is the challenge. For my server, and for its purpose, I am choosing the following:

iptables - firewall

Fail2Ban - set to permanently ban on three log in failure attempts. Did this by changing the default tries of 5 to 3 and ban timeout to 0 (infinite). I use a very strong password, so this is really overkill. Mostly it annoys me that failed log in attempts are clogging up my log files. I use sudo, so root and ssh users have log in disabled. There are hardly any other users that are attempted for break in attempts.

Obscurity - changed the ssh port from default to some other large number under the max of 65,535.

Tripwire - let's me know who gains access and when. If they weren't supposed to, I can revert to backup and change passwords.

md5 checksums - get a clean list of check sums after all software is installed, and before connecting to the internet and save to USB or remote location. This way I can periodically check the installed applications to verify their integrity in case crafty intruder managed to sneak in and root the server. Looks like rkhunter might have some functionality to automate this, so I'll look into that.

ssh - using private key with pass phrase and disabled password log in capability for all users.

I'm also considering one other suggestions posted on another site, use separate server to monitor network traffic. Since Tripwire could be compromised if the server is broken into, this would be another option for a more secure way to monitor connections. It however is crossing the line into ultra paranoid, and still, seems like an interesting thing to explore.

Also, an idea I'm toying with is creating a simple script that would monitor the 'who' output, and trigger a timer that when expired will disable all NICs. The script would have to be disabled shortly after logging in to prevent the server from being cut off from the outside world. Crafty intruder would have to have first hand knowledge of such a script in order to defeat it. Again, major overkill, but simple to implement, and easy to maintain.

Finally, if you made it this far, then you will certainly enjoy the following article, which chronicles the adventures of a security expert who sets up servers with the goal of baiting would be crafty intruders to analyze what they do. In this case it is a person or organization that utilizes servers to effect DOS attacks:

http://draios.com/fishing-for-hackers/
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Ebury - how check system? bigmen58 Linux - Security 30 01-28-2014 06:11 PM
/home folder not exporting from Sever A (Ubuntu 11.04) to Server B(Ubuntu 12.04) Aholton Linux - Server 1 06-28-2012 10:07 PM
Starting sshd: /etc/init.d/sshd: line 113: /usr/sbin/sshd: Permission denied sumanc Linux - Server 5 03-28-2008 04:59 AM
FC4-Starting sshd: Privilege separation user sshd does not exist FAILED kiranherekar Fedora 5 12-29-2005 02:22 PM


All times are GMT -5. The time now is 03:34 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration