My Server (Deb5 and Plesk10) is involved (causing) in brute force attacks
Linux - SecurityThis forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
My Server (Deb5 and Plesk10) is involved (causing) in brute force attacks
I am ashamed that I am causing other people troubles, but apparantly my server is involved in attacking the servers of other people.
I have to admit that I am not too familiar with using a CLI, or Linux for that matter, but I have a Debian server running under Plesk 10, which is colocated.
Now I have received messages from the datacenterm which state that my server is involved in brute force attacks.
The messages show a lot of lines like this:
Code:
May 5 01:15:47 hostname-victim-server sshd[18807]: Failed password for root from xxx.xxx.xxx.xxx (My IP) port 47971 ssh2
The only suggestion I get from my hoster is to back up all domains and re-install the machine.
I want to resolve this asap, but do not agree with that action for two reasons: the machine just had a fresh re-install 2 months ago, so if it is a flaw in the OS, I will get the same flaw back, and if it is not OS related but due to a domain, I will get the problem back by putting back the backed-up domains.
But now I'm stuck: what steps should I follow to try and find the cause of this evil and make sure that my machine will not bother other machines anymore?
I realize that this probably will be a steep learning-curve, but please bare with me and help me to resolve this.
What have I done so far?
1) There are a number of live sites on this server, either running WordPress or Joomla, I have made sure they are all updated to the latest release.
2) I have manually looked at the source code of the index-files of those sites, haven't seen anything strange, like redirects.
3) I have used online scanners to check all sites for malware, all have been reported back to be clean.
4) I have run the Plesk-version of RKhunter, and that gives me certain warnings which I cannot (or do not) understand:
Code:
Performing trojan specific checks
Checking for enabled xinetd services [ Warning ]
Code:
Performing group and account checks
Checking for passwd file [ Found ]
Checking for root equivalent (UID 0) accounts [ None found ]
Checking for passwordless accounts [ None found ]
Checking for passwd file changes [ Warning ]
Checking for group file changes [ None found ]
Checking root account shell history files [ OK ]
Code:
Performing system configuration file checks
Checking for SSH configuration file [ Found ]
Checking if SSH root access is allowed [ Warning ]
Checking if SSH protocol v1 is allowed [ Not allowed ]
Checking for running syslog daemon [ Found ]
Checking for syslog configuration file [ Found ]
Checking if syslog remote logging is allowed [ Not allowed ]
Performing filesystem checks
Checking /dev for suspicious file types [ None found ]
Checking for hidden files and directories [ Warning ]
[Press <ENTER> to continue]
Checking application versions...
Checking version of GnuPG [ Warning ]
Checking version of Bind DNS [ OK ]
Checking version of OpenSSL [ Warning ]
Checking version of PHP [ Warning ]
Checking version of Procmail MTA [ OK ]
Checking version of ProFTPd [ Skipped ]
Checking version of OpenSSH [ Warning ]
I received the first report of these attempts about a week ago and immediately changed the Plesk/SSH password to a 200bit password generated with KeePass, hoping that would keep out any evildoers, but sadly enough that wasn't the simple solution I had been hoping for
I see what you are getting at, but this is my situation:
1) I call my colo-hoster and ask them to send remote-hands to my machine and do a re-install
2) They do that, reinstall Deb5 and Plesk10, including the most recent updates
And at that point I am back at the exact same situtation as I am right now. Like I mentioned before, they did this re-install 2 month ago.
So if I just do another re-install, without knowing what went wrong the first time, I'm in the same place. I put back the backups from the sites that are running right now, and all factors are the back in place, waiting for another hack to happen.
You see why I first want to learn what has gone wrong?
@MartinM: Welcome to LQ-Security. I wish to applaud you on the approach you are taking and that you recognize the seriousness of the situation and the importance of performing an in depth investigation as to the root cause of the problem. In the case of a root level compromise, ultimately part of the recovery is a wipe and re-install of the operating system. However, this should never be amongst the fist things you consider and this should not have been brought up like it was as it is not how we work at LQ-Security.
Your thoughts on checking the "Intruder Detection Checklist" is absolutely one of the best things you can do. Here is a link for your reference. Before you begin, I would like to suggest that you isolate the machine as best as you can. Since this is a co-located machine, you should put up the firewall (iptables) and block all traffic, except for SSH from a trusted IP that you will use. In this particular case, I would also highly recommend that you verify the integrity of the SSH binary application against the ones contained in the Debian distribution for your revision. Given the nature of the incident and your lack of physical contact, it is imperative that you ascertain the integrity of the SSH application before you proceed.
Second, as you are probably becoming aware, RKHunter is prone to false positives. As such, if you are going to use it, you need to educate yourself on its intricacies. In this particular case, it is giving you some initial places to look. One thing about RKHunter is that it can detect that files have been modified. It can't tell you if the modifications are legitimate. Lets examine the warnings:
Checking for passwd file changes [ Warning ] - this could indicate that the user accounts have changed. This could be a result of your modifications or the intruder. Examine this file carefully.
Checking if SSH root access is allowed - this means that your SSH configuration allows direct root login. This is discouraged. Coupled with password authentication, you may be subject to brute force root password attack and compromise.
Checking for hidden files and directories - normally an intruder tries to hide their tracks by putting hidden binaries in strange places, like /tmp and /dev. This is one of the task items from the cert check list. Unfortunately, this is also a very common false positive.
Checking for version of .... This indicates that your versions are likely out of date, or at least not the most recent. Unfortunately, Debian runs behind most other distributions so you will need to weight this carefully.
After you have secured the machine and reviewed the check list, please obtain the output of the following commands and post it so that we can review. You will probably need to attach the capture as files which you can do under the "go advanced" tab.
Code:
ps axfwwwe 2>&1; netstat -anpe 2>&1; lsof -Pwln 2>&1; who 2>&1; last 2>&1;
This will provide the list of processes that are running, the connections open both for inter process communication and internet sockets. The idea is that we will look for suspicious connections and applications that could be responsible. Once you have completed this step and we are able to analyze the output, we will determine the next steps.
Once compromised you can never again be sure of the security of the machine until it is re-installed.
Re-install.
THEN take steps to secure that fresh install as best you possibly can.
Do NOT reinstall until the nature of the compromise is discovered (who, what, when, where, why, how, at the very least). Reinstalling without investigating would more than likely put you in a vulnerable position again. Reinstalling without investigating will erase any auditing information that the system may have recorded.
This type of response is considered hair-trigger and is frowned upon in these forums.
OP, Noway2 is pointing the OP in the proper direction. Follow his instruction.
EDIT: and I see that the OP has already experienced the round-robin of reinstalling and being re-compromised...
HAHAHA! I'm alive and well. Family and work life has been too crazy for me to participate in ##slackware chat, so I try to contribute here as much as possible (doing my part ). It's easier for me to insert a plug here and there in the forum. I do miss IRCing, though...
Thanks for the help so far, it's getting clearer and clearer that I was at least thinking in the right direction. Now it's time to get going in the right direction too
About isolating my machine: (sorry, I'm a GUI-man, not a CLI-man ), can I do this by following these instructions from Plesk? -->
Quote:
To alleviate security concerns, you may want to restrict administrative access to your control panel from specific IP addresses.
To allow administrative access to the Panel only from specific IP addresses or networks:
Go to Settings > Restrict Administrative Access (in the Security group).
Click Add New Network and specify the required IP addresses. Click OK.
To specify subnets, you can use wildcard symbols (*) and subnet masks.
Select the Denied from the networks that are not listed option, and click Set. When prompted to confirm the operation, click OK.
And if so, trying to think ahead....at home I have a connection with a dynamic IP, not that it changes every week or even month, but when it changes this would mean I cannot connect to the machine myself anymore, correct?
Is there a workaround for that? (My common sense says no, except maybe for adding a dedicated IP from the support-desk of the colo-hoster, so I can call them and ask them to add my new IP if this would happen)
I'm afraid the process is still going on at this moment, after using that command that Noway2 gave me, I notice this:
Even though I am not familiar with the ins and outs of this, it looks to me as if it is telling me that my server is using all kind of child processes to scan another machines' SSH port.
Addition 2:
I've got a feeling that this is the root of my evil, could I be correct? :
I see this repeated a huge number of times, also combined with the following terminology " \_ ./ssh-scan" and the IP is a Romanian IP, and I am quite sure that none of my Dutch friends who have access to my server (5 in all and all personal acquaintances) live there ....
Last edited by MartinM; 05-06-2011 at 10:06 AM.
Reason: additional information
And if so, trying to think ahead....at home I have a connection with a dynamic IP, not that it changes every week or even month, but when it changes this would mean I cannot connect to the machine myself anymore, correct?
Is there a workaround for that? (My common sense says no, except maybe for adding a dedicated IP from the support-desk of the colo-hoster, so I can call them and ask them to add my new IP if this would happen)
Assuming that you are going ahead with the suggestion of, effectively, white listing your IP address in your iptables set-up, and also assuming that your home ISP can tell you what address range(s) they have allocated to them, you could whitelist (let through) that whole range in iptables. Not hard, but do you have any iptables expertise?
One quibble that I would make with Noway's otherwise excellent post is that I would have stated
Quote:
this means that your SSH configuration allows direct root login. This is discouraged.
more forcefully.
Don't allow any direct root logins
You have to understand that there are people out there who will try to break in. You may not like it, you may think that it shouldn't be, and you may even say 'why me?', but it is a fact of life. If someone can keep trying and trying to break in (and root is 'the keys to the kingdom', so that is what they will prefer to get), then, eventually, they will break in (or, the world will come to an end, whichever happens first). You have to do something more 'sophisticated' (or complex) than this to have a chance, in the longer term.
For ssh, there is a summary here. The good news is that there are plenty of alternatives; the bad news is that you've got to choose one, or more, and actually put it in to action.
At one time, it seemed to be only the guys who chose the more painfully obvious passwords who got cracked; the days when just a slightly strong password keeps you out of trouble are gone and don't look like coming back.
Yes, those indicate that you have a running process that is currently performing SSH scans. You can see that it is claiming that the user is 0 (root), but that there it appears that the inode and process ID information are bogus. For example, I am using a machine at work right now to SSH into my own home PC. Running the command netstate -pane displays the connection along with the SSH process, the inode, and the ssh PID for the connection. My immediate suspicion is that you have a rogue process with root level privilege.
I believe it is imperative for you to get a firewall up on this machine. Do not reboot it, etc. Try to disturb as little as possible. Your objective is to catch this process. Securing the system via the command line should be easy, except for your dynamic IP (more on that in a second). I think trying to use a GUI may be more complicated due to the limitations inherent in a gui; limited to what the designer decides to offer.
You can use IP tables commands to stop the traffic. Iptables rules work like a waterfall, so order is important. To shut down all but SSH traffic from your known source the comamnds would be as follows:
iptables -A INPUT -p tcp -m tcp -s <your-ip> --dport 22 -j ACCEPT
iptables -A INPUT -j DROP
You will also want to shut down port 22 on the output. Your SSH connection shouldn't use port 22 outbound
iptables -A OUTPUT ! -d <your ip> -j DROP # Blocks all packets NOT going to your IP
These commands must be run as root! The first command adds a rule to the inbound packet list that allows port 22 TCP (ssh traffic) from your address. The second line blocks all other inbound connections. The third line will block all output bound traffic that is not back to you. This will stop the scanning, but leave the process running and continuing to attempt to run.
In this particular case, you will want to use a range of IP addresses associated with your provider. Chances are you can grab a whole network range and stop the trouble, at least stop the bleeding and give yourself room to breath.
Once you get the system secure, you need to start tracking down this file. Proceed with the full netstat command and the, ps and lsof. Also from the Cert list there are instructions on how to look for files with setuid and setguid. You will want to look for those.
Regarding your addition number 2: Yes, I believe you may have found at least a tendril of the evil root. It does look like your root account has been hijacked.
Specifically, they are running a screen process, which will let them disconnect and keep the processes running in the background. The user root has been commandeered, the path has been set, and the command logging is going in the bit bucket.
Edit: yes, RSA authentication would be best. Password authentication set to no. However - I will REPEAT - YOUR SSH MAY BE COMPROMISED AND MAY LIE TO YOU. USING THE SSH AND SECURING BY RSA KEY MAY BE FUTILE! Iptables / netfilter has a little better chance of being ok since this is compiled into the kernel rather than being an application.
In this particular case, you will want to use a range of IP addresses associated with your provider. Chances are you can grab a whole network range and stop the trouble, at least stop the bleeding and give yourself room to breath.
That is going to be a bit hard, since I know for a fact that my ISP uses for example 77.248.xx.xx, 77.251.xx.xx, 62.163.xx.xx, and a dozen other IP-blocks, so I never really know what IP to expect....
Quote:
Iptables / netfilter has a little better chance of being ok since this is compiled into the kernel rather than being an application.
Ok, so I do not have to install anything, this will not conflict with the current Plesk setup and I just have to perform the commands you gave me, correct? (And yes, in the right order so I will not shut myself out prematurely )
Addition:
I have followed the steps to update the Iptables. Does this mean that at least I have time to breathe now and the outsideworld will not be bothered by me for the time being?
I have attached the results of netstat and ps, the results of lsof was too big so I need to find another solution for that...
Last edited by MartinM; 05-06-2011 at 12:26 PM.
Reason: Additions
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
My Server (Deb5 and Plesk10) is involved (causing) in brute force attacks
