LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices


Reply
  Search this Thread
Old 05-06-2011, 05:01 AM   #1
MartinM
Member
 
Registered: May 2011
Location: the Netherlands
Distribution: Debian Squeeze
Posts: 39

Rep: Reputation: 2
Unhappy My Server (Deb5 and Plesk10) is involved (causing) in brute force attacks


I am ashamed that I am causing other people troubles, but apparantly my server is involved in attacking the servers of other people.

I have to admit that I am not too familiar with using a CLI, or Linux for that matter, but I have a Debian server running under Plesk 10, which is colocated.

Now I have received messages from the datacenterm which state that my server is involved in brute force attacks.

The messages show a lot of lines like this:

Code:
May  5 01:15:47 hostname-victim-server sshd[18807]: Failed password for root from xxx.xxx.xxx.xxx (My IP) port 47971 ssh2
The only suggestion I get from my hoster is to back up all domains and re-install the machine.

I want to resolve this asap, but do not agree with that action for two reasons: the machine just had a fresh re-install 2 months ago, so if it is a flaw in the OS, I will get the same flaw back, and if it is not OS related but due to a domain, I will get the problem back by putting back the backed-up domains.

But now I'm stuck: what steps should I follow to try and find the cause of this evil and make sure that my machine will not bother other machines anymore?

I realize that this probably will be a steep learning-curve, but please bare with me and help me to resolve this.

What have I done so far?

1) There are a number of live sites on this server, either running WordPress or Joomla, I have made sure they are all updated to the latest release.

2) I have manually looked at the source code of the index-files of those sites, haven't seen anything strange, like redirects.

3) I have used online scanners to check all sites for malware, all have been reported back to be clean.

4) I have run the Plesk-version of RKhunter, and that gives me certain warnings which I cannot (or do not) understand:

Code:
Performing trojan specific checks
    Checking for enabled xinetd services                     [ Warning ]
Code:
Performing group and account checks
    Checking for passwd file                                 [ Found ]
    Checking for root equivalent (UID 0) accounts            [ None found ]
    Checking for passwordless accounts                       [ None found ]
    Checking for passwd file changes                         [ Warning ]
    Checking for group file changes                          [ None found ]
    Checking root account shell history files                [ OK ]

Code:
Performing system configuration file checks
    Checking for SSH configuration file                      [ Found ]
    Checking if SSH root access is allowed                   [ Warning ]
    Checking if SSH protocol v1 is allowed                   [ Not allowed ]
    Checking for running syslog daemon                       [ Found ]
    Checking for syslog configuration file                   [ Found ]
    Checking if syslog remote logging is allowed             [ Not allowed ]

  Performing filesystem checks
    Checking /dev for suspicious file types                  [ None found ]
    Checking for hidden files and directories                [ Warning ]

[Press <ENTER> to continue]

Checking application versions...

    Checking version of GnuPG                                [ Warning ]
    Checking version of Bind DNS                             [ OK ]
    Checking version of OpenSSL                              [ Warning ]
    Checking version of PHP                                  [ Warning ]
    Checking version of Procmail MTA                         [ OK ]
    Checking version of ProFTPd                              [ Skipped ]
    Checking version of OpenSSH                              [ Warning ]
I received the first report of these attempts about a week ago and immediately changed the Plesk/SSH password to a 200bit password generated with KeePass, hoping that would keep out any evildoers, but sadly enough that wasn't the simple solution I had been hoping for
 
Old 05-06-2011, 05:05 AM   #2
zordrak
Member
 
Registered: Feb 2008
Distribution: Slackware
Posts: 595

Rep: Reputation: 115Reputation: 115
Bottom line:

Once compromised you can never again be sure of the security of the machine until it is re-installed.

Re-install.

THEN take steps to secure that fresh install as best you possibly can.
 
0 members found this post helpful.
Old 05-06-2011, 05:11 AM   #3
MartinM
Member
 
Registered: May 2011
Location: the Netherlands
Distribution: Debian Squeeze
Posts: 39

Original Poster
Rep: Reputation: 2
Hi Zordrak,

tnx for the speedy response.

I see what you are getting at, but this is my situation:

1) I call my colo-hoster and ask them to send remote-hands to my machine and do a re-install
2) They do that, reinstall Deb5 and Plesk10, including the most recent updates

And at that point I am back at the exact same situtation as I am right now. Like I mentioned before, they did this re-install 2 month ago.
So if I just do another re-install, without knowing what went wrong the first time, I'm in the same place. I put back the backups from the sites that are running right now, and all factors are the back in place, waiting for another hack to happen.

You see why I first want to learn what has gone wrong?

Last edited by MartinM; 05-06-2011 at 05:18 AM.
 
Old 05-06-2011, 05:32 AM   #4
zordrak
Member
 
Registered: Feb 2008
Distribution: Slackware
Posts: 595

Rep: Reputation: 115Reputation: 115
I understand.

You probably want to start here:

RKHunter FAQ
 
1 members found this post helpful.
Old 05-06-2011, 07:03 AM   #5
MartinM
Member
 
Registered: May 2011
Location: the Netherlands
Distribution: Debian Squeeze
Posts: 39

Original Poster
Rep: Reputation: 2
Quote:
Originally Posted by zordrak View Post
I understand.

You probably want to start here:

RKHunter FAQ
Thanks, I'll dive right into it, including the links provided in that How-To about the "Intruder Detection Checklist".

That should keep me busy for today
 
Old 05-06-2011, 07:59 AM   #6
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Gentoo
Posts: 2,125

Rep: Reputation: 780Reputation: 780Reputation: 780Reputation: 780Reputation: 780Reputation: 780Reputation: 780
@MartinM: Welcome to LQ-Security. I wish to applaud you on the approach you are taking and that you recognize the seriousness of the situation and the importance of performing an in depth investigation as to the root cause of the problem. In the case of a root level compromise, ultimately part of the recovery is a wipe and re-install of the operating system. However, this should never be amongst the fist things you consider and this should not have been brought up like it was as it is not how we work at LQ-Security.

Your thoughts on checking the "Intruder Detection Checklist" is absolutely one of the best things you can do. Here is a link for your reference. Before you begin, I would like to suggest that you isolate the machine as best as you can. Since this is a co-located machine, you should put up the firewall (iptables) and block all traffic, except for SSH from a trusted IP that you will use. In this particular case, I would also highly recommend that you verify the integrity of the SSH binary application against the ones contained in the Debian distribution for your revision. Given the nature of the incident and your lack of physical contact, it is imperative that you ascertain the integrity of the SSH application before you proceed.

Second, as you are probably becoming aware, RKHunter is prone to false positives. As such, if you are going to use it, you need to educate yourself on its intricacies. In this particular case, it is giving you some initial places to look. One thing about RKHunter is that it can detect that files have been modified. It can't tell you if the modifications are legitimate. Lets examine the warnings:

Checking for passwd file changes [ Warning ] - this could indicate that the user accounts have changed. This could be a result of your modifications or the intruder. Examine this file carefully.
Checking if SSH root access is allowed - this means that your SSH configuration allows direct root login. This is discouraged. Coupled with password authentication, you may be subject to brute force root password attack and compromise.
Checking for hidden files and directories - normally an intruder tries to hide their tracks by putting hidden binaries in strange places, like /tmp and /dev. This is one of the task items from the cert check list. Unfortunately, this is also a very common false positive.
Checking for version of .... This indicates that your versions are likely out of date, or at least not the most recent. Unfortunately, Debian runs behind most other distributions so you will need to weight this carefully.

After you have secured the machine and reviewed the check list, please obtain the output of the following commands and post it so that we can review. You will probably need to attach the capture as files which you can do under the "go advanced" tab.
Code:
ps axfwwwe 2>&1; netstat -anpe 2>&1; lsof -Pwln 2>&1; who 2>&1; last 2>&1;
This will provide the list of processes that are running, the connections open both for inter process communication and internet sockets. The idea is that we will look for suspicious connections and applications that could be responsible. Once you have completed this step and we are able to analyze the output, we will determine the next steps.
 
5 members found this post helpful.
Old 05-06-2011, 08:22 AM   #7
unixfool
Member
 
Registered: May 2005
Location: Northern VA
Distribution: Slackware, Ubuntu, FreeBSD, OpenBSD, OS X
Posts: 782
Blog Entries: 8

Rep: Reputation: 157Reputation: 157
Quote:
Originally Posted by zordrak View Post
Bottom line:

Once compromised you can never again be sure of the security of the machine until it is re-installed.

Re-install.

THEN take steps to secure that fresh install as best you possibly can.
Do NOT reinstall until the nature of the compromise is discovered (who, what, when, where, why, how, at the very least). Reinstalling without investigating would more than likely put you in a vulnerable position again. Reinstalling without investigating will erase any auditing information that the system may have recorded.

This type of response is considered hair-trigger and is frowned upon in these forums.

OP, Noway2 is pointing the OP in the proper direction. Follow his instruction.

EDIT: and I see that the OP has already experienced the round-robin of reinstalling and being re-compromised...

Last edited by unixfool; 05-06-2011 at 08:24 AM.
 
4 members found this post helpful.
Old 05-06-2011, 08:27 AM   #8
zordrak
Member
 
Registered: Feb 2008
Distribution: Slackware
Posts: 595

Rep: Reputation: 115Reputation: 115
My bad. Pulled post from zero-reply and figured user's primary concern was fix rather than learn. I know different now.

@uf you're not dead?!
 
Old 05-06-2011, 09:36 AM   #9
unixfool
Member
 
Registered: May 2005
Location: Northern VA
Distribution: Slackware, Ubuntu, FreeBSD, OpenBSD, OS X
Posts: 782
Blog Entries: 8

Rep: Reputation: 157Reputation: 157
Quote:
Originally Posted by zordrak View Post
@uf you're not dead?!
HAHAHA! I'm alive and well. Family and work life has been too crazy for me to participate in ##slackware chat, so I try to contribute here as much as possible (doing my part ). It's easier for me to insert a plug here and there in the forum. I do miss IRCing, though...

Last edited by unixfool; 05-06-2011 at 09:38 AM.
 
Old 05-06-2011, 09:37 AM   #10
MartinM
Member
 
Registered: May 2011
Location: the Netherlands
Distribution: Debian Squeeze
Posts: 39

Original Poster
Rep: Reputation: 2
Thanks for the help so far, it's getting clearer and clearer that I was at least thinking in the right direction. Now it's time to get going in the right direction too

About isolating my machine: (sorry, I'm a GUI-man, not a CLI-man ), can I do this by following these instructions from Plesk? -->

Quote:
To alleviate security concerns, you may want to restrict administrative access to your control panel from specific IP addresses.

To allow administrative access to the Panel only from specific IP addresses or networks:
Go to Settings > Restrict Administrative Access (in the Security group).
Click Add New Network and specify the required IP addresses. Click OK.
To specify subnets, you can use wildcard symbols (*) and subnet masks.
Select the Denied from the networks that are not listed option, and click Set. When prompted to confirm the operation, click OK.
And if so, trying to think ahead....at home I have a connection with a dynamic IP, not that it changes every week or even month, but when it changes this would mean I cannot connect to the machine myself anymore, correct?
Is there a workaround for that? (My common sense says no, except maybe for adding a dedicated IP from the support-desk of the colo-hoster, so I can call them and ask them to add my new IP if this would happen)

I'm afraid the process is still going on at this moment, after using that command that Noway2 gave me, I notice this:


tcp 0 0 xx.xx.xxx.xx:43451 93.174.3.211:22 ESTABLISHED 0 53807315 -
tcp 0 0 xx.xx.xxx.xx:48043 93.174.3.208:22 TIME_WAIT 0 0 -
tcp 0 0 xx.xx.xxx.xx:52970 93.174.2.59:22 TIME_WAIT 0 0 -
tcp 0 0 xx.xx.xxx.xx:43357 93.174.2.152:22 TIME_WAIT 0 0 -
tcp 0 0 xx.xx.xxx.xx:37650 93.174.187.87:22 TIME_WAIT 0 0 -
tcp 0 0 xx.xx.xxx.xx:46658 93.174.188.110:22 TIME_WAIT 0 0 -
tcp 0 0 xx.xx.xxx.xx:36548 93.174.2.105:22 TIME_WAIT 0 0 -
tcp 0 0 xx.xx.xxx.xx:36974 93.174.2.45:22 TIME_WAIT 0 0 -
tcp 0 0 xx.xx.xxx.xx:43350 93.174.2.153:22 TIME_WAIT 0 0 -
tcp 0 0 xx.xx.xxx.xx:50934 93.174.2.157:22 TIME_WAIT 0 0 -
tcp 0 0 xx.xx.xxx.xx:34867 93.174.2.141:22 TIME_WAIT 0 0 -
tcp 0 0 xx.xx.xxx.xx:44091 93.174.3.200:22 TIME_WAIT 0 0 -
tcp 0 0 xx.xx.xxx.xx:42607 93.174.3.208:22 TIME_WAIT 0 0 -
tcp 0 0 xx.xx.xxx.xx:42374 93.174.2.92:22 TIME_WAIT 0 0 -

Even though I am not familiar with the ins and outs of this, it looks to me as if it is telling me that my server is using all kind of child processes to scan another machines' SSH port.

Addition 2:

I've got a feeling that this is the root of my evil, could I be correct? :

Quote:
4268 ? Ss 0:14 SCREEN TERM=xterm SHELL=/bin/bash SSH_CLIENT=::ffff:89.33.63.17 57952 22 SSH_TTY=/dev/pts/0 USER=root MAIL=/var/mail/root PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PWD=/usr/games/go PS1=\h:\w\$ SHLVL=1 HOME=/root LOGNAME=root SSH_CONNECTION=::ffff:89.33.63.17 57952 ::ffff:85.17.219.22 22 HISTFILE=/dev/null _=/usr/bin/screen OLDPWD=/root
I see this repeated a huge number of times, also combined with the following terminology " \_ ./ssh-scan" and the IP is a Romanian IP, and I am quite sure that none of my Dutch friends who have access to my server (5 in all and all personal acquaintances) live there ....

Last edited by MartinM; 05-06-2011 at 10:06 AM. Reason: additional information
 
Old 05-06-2011, 10:08 AM   #11
salasi
Senior Member
 
Registered: Jul 2007
Location: Directly above centre of the earth, UK
Distribution: SuSE, plus some hopping
Posts: 4,070

Rep: Reputation: 897Reputation: 897Reputation: 897Reputation: 897Reputation: 897Reputation: 897Reputation: 897
Quote:
Originally Posted by MartinM View Post
And if so, trying to think ahead....at home I have a connection with a dynamic IP, not that it changes every week or even month, but when it changes this would mean I cannot connect to the machine myself anymore, correct?
Is there a workaround for that? (My common sense says no, except maybe for adding a dedicated IP from the support-desk of the colo-hoster, so I can call them and ask them to add my new IP if this would happen)
Assuming that you are going ahead with the suggestion of, effectively, white listing your IP address in your iptables set-up, and also assuming that your home ISP can tell you what address range(s) they have allocated to them, you could whitelist (let through) that whole range in iptables. Not hard, but do you have any iptables expertise?

One quibble that I would make with Noway's otherwise excellent post is that I would have stated
Quote:
this means that your SSH configuration allows direct root login. This is discouraged.
more forcefully.

Don't allow any direct root logins

You have to understand that there are people out there who will try to break in. You may not like it, you may think that it shouldn't be, and you may even say 'why me?', but it is a fact of life. If someone can keep trying and trying to break in (and root is 'the keys to the kingdom', so that is what they will prefer to get), then, eventually, they will break in (or, the world will come to an end, whichever happens first). You have to do something more 'sophisticated' (or complex) than this to have a chance, in the longer term.

For ssh, there is a summary here. The good news is that there are plenty of alternatives; the bad news is that you've got to choose one, or more, and actually put it in to action.

At one time, it seemed to be only the guys who chose the more painfully obvious passwords who got cracked; the days when just a slightly strong password keeps you out of trouble are gone and don't look like coming back.
 
3 members found this post helpful.
Old 05-06-2011, 10:19 AM   #12
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Gentoo
Posts: 2,125

Rep: Reputation: 780Reputation: 780Reputation: 780Reputation: 780Reputation: 780Reputation: 780Reputation: 780
Yes, those indicate that you have a running process that is currently performing SSH scans. You can see that it is claiming that the user is 0 (root), but that there it appears that the inode and process ID information are bogus. For example, I am using a machine at work right now to SSH into my own home PC. Running the command netstate -pane displays the connection along with the SSH process, the inode, and the ssh PID for the connection. My immediate suspicion is that you have a rogue process with root level privilege.

I believe it is imperative for you to get a firewall up on this machine. Do not reboot it, etc. Try to disturb as little as possible. Your objective is to catch this process. Securing the system via the command line should be easy, except for your dynamic IP (more on that in a second). I think trying to use a GUI may be more complicated due to the limitations inherent in a gui; limited to what the designer decides to offer.

You can use IP tables commands to stop the traffic. Iptables rules work like a waterfall, so order is important. To shut down all but SSH traffic from your known source the comamnds would be as follows:

iptables -A INPUT -p tcp -m tcp -s <your-ip> --dport 22 -j ACCEPT
iptables -A INPUT -j DROP

You will also want to shut down port 22 on the output. Your SSH connection shouldn't use port 22 outbound
iptables -A OUTPUT ! -d <your ip> -j DROP # Blocks all packets NOT going to your IP

These commands must be run as root! The first command adds a rule to the inbound packet list that allows port 22 TCP (ssh traffic) from your address. The second line blocks all other inbound connections. The third line will block all output bound traffic that is not back to you. This will stop the scanning, but leave the process running and continuing to attempt to run.

In this particular case, you will want to use a range of IP addresses associated with your provider. Chances are you can grab a whole network range and stop the trouble, at least stop the bleeding and give yourself room to breath.

Once you get the system secure, you need to start tracking down this file. Proceed with the full netstat command and the, ps and lsof. Also from the Cert list there are instructions on how to look for files with setuid and setguid. You will want to look for those.
 
3 members found this post helpful.
Old 05-06-2011, 10:26 AM   #13
MartinM
Member
 
Registered: May 2011
Location: the Netherlands
Distribution: Debian Squeeze
Posts: 39

Original Poster
Rep: Reputation: 2
Ok, point taken.

Since I am the only one who needs SSH, I'd say that using RSA authentication would be a good choice, maybe even the best in my case, correct?
 
Old 05-06-2011, 10:26 AM   #14
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Gentoo
Posts: 2,125

Rep: Reputation: 780Reputation: 780Reputation: 780Reputation: 780Reputation: 780Reputation: 780Reputation: 780
Regarding your addition number 2: Yes, I believe you may have found at least a tendril of the evil root. It does look like your root account has been hijacked.
Specifically, they are running a screen process, which will let them disconnect and keep the processes running in the background. The user root has been commandeered, the path has been set, and the command logging is going in the bit bucket.

Edit: yes, RSA authentication would be best. Password authentication set to no. However - I will REPEAT - YOUR SSH MAY BE COMPROMISED AND MAY LIE TO YOU. USING THE SSH AND SECURING BY RSA KEY MAY BE FUTILE! Iptables / netfilter has a little better chance of being ok since this is compiled into the kernel rather than being an application.

Last edited by Noway2; 05-06-2011 at 10:29 AM.
 
2 members found this post helpful.
Old 05-06-2011, 10:34 AM   #15
MartinM
Member
 
Registered: May 2011
Location: the Netherlands
Distribution: Debian Squeeze
Posts: 39

Original Poster
Rep: Reputation: 2
Quote:
Originally Posted by Noway2 View Post
In this particular case, you will want to use a range of IP addresses associated with your provider. Chances are you can grab a whole network range and stop the trouble, at least stop the bleeding and give yourself room to breath.
That is going to be a bit hard, since I know for a fact that my ISP uses for example 77.248.xx.xx, 77.251.xx.xx, 62.163.xx.xx, and a dozen other IP-blocks, so I never really know what IP to expect....

Quote:
Iptables / netfilter has a little better chance of being ok since this is compiled into the kernel rather than being an application.
Ok, so I do not have to install anything, this will not conflict with the current Plesk setup and I just have to perform the commands you gave me, correct? (And yes, in the right order so I will not shut myself out prematurely )

Addition:
I have followed the steps to update the Iptables. Does this mean that at least I have time to breathe now and the outsideworld will not be bothered by me for the time being?

I have attached the results of netstat and ps, the results of lsof was too big so I need to find another solution for that...
Attached Files
File Type: pdf netstat.pdf (28.7 KB, 10 views)
File Type: pdf ps.pdf (11.1 KB, 17 views)

Last edited by MartinM; 05-06-2011 at 12:26 PM. Reason: Additions
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[iptables] How to block brute force attacks? littlebigman Linux - Software 2 04-05-2011 04:48 AM
[SOLVED] Server receiving a lot of brute force SSH attacks the182guy Linux - Newbie 6 10-16-2009 08:27 AM
[SOLVED] MySql-ban brute force attacks? qwertyjjj Linux - Software 3 08-10-2009 05:28 AM
Does anyone know if guardian can be set to block brute force attacks and only brute f abefroman Linux - Software 2 06-05-2008 10:55 AM
Question on Brute Force Attacks Mad Mike Linux - Security 4 10-16-2006 10:25 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Security

All times are GMT -5. The time now is 12:44 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration