Linux - SecurityThis forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
I am having a constant problem with my server being targeted by a botnet to send spam. Someone, or something, is uploading/creating a file or the server which a botnet is then accessing to send out spam.
The PHP file being created/uploaded is obfuscated so it is not clear what it does from looking at it, but as soon as its on the server and accessed it sends email out.
Fortunately the domain it is targeting is not used for sending emails, so I have configured Exim to freeze the emails.
The below log is when the file is created - it appears to be someone with firefox checking the existence of the hack, finding it missing (404) and uploading/recreating it and then testing again (200).
Approximately ten minutes after this, the bot net starts accessing it:
Code:
./webserver:67:109.205.136.19 - - [03/Dec/2014:00:55:09 +1000] "POST //connect/resources/model.php HTTP/1.1" 200 65 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.7.6)"
./webserver:68:109.205.136.19 - - [03/Dec/2014:00:55:59 +1000] "POST //connect/resources/model.php HTTP/1.1" 200 184 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.7.6)"
./webserver:69:109.205.136.19 - - [03/Dec/2014:00:56:51 +1000] "POST //connect/resources/model.php HTTP/1.1" 200 65 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.7.6)"
./webserver:70:109.205.136.19 - - [03/Dec/2014:00:57:45 +1000] "POST //connect/resources/model.php HTTP/1.1" 200 65 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.7.6)"
./webserver:71:109.205.136.19 - - [03/Dec/2014:00:58:37 +1000] "POST //connect/resources/model.php HTTP/1.1" 200 65 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.7.6)"
./webserver:72:109.205.136.19 - - [03/Dec/2014:00:59:26 +1000] "POST //connect/resources/model.php HTTP/1.1" 200 65 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.7.6)"
./webserver:73:109.205.136.19 - - [03/Dec/2014:01:00:18 +1000] "POST //connect/resources/model.php HTTP/1.1" 200 65 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.7.6)"
./webserver:74:69.195.124.107 - - [03/Dec/2014:01:01:06 +1000] "POST //connect/resources/model.php HTTP/1.1" 200 296 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.7.6)"
./webserver:75:109.205.136.19 - - [03/Dec/2014:01:01:10 +1000] "POST //connect/resources/model.php HTTP/1.1" 200 65 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.7.6)"
./webserver:76:69.195.124.107 - - [03/Dec/2014:01:02:01 +1000] "POST //connect/resources/model.php HTTP/1.1" 200 65 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.7.6)"
./webserver:77:109.205.136.19 - - [03/Dec/2014:01:02:02 +1000] "POST //connect/resources/model.php HTTP/1.1" 200 65 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.7.6)"
./webserver:78:109.205.136.19 - - [03/Dec/2014:01:02:56 +1000] "POST //connect/resources/model.php HTTP/1.1" 200 65 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.7.6)"
./webserver:79:69.195.124.107 - - [03/Dec/2014:01:02:56 +1000] "POST //connect/resources/model.php HTTP/1.1" 200 65 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.7.6)"
./webserver:80:69.89.31.134 - - [03/Dec/2014:01:03:07 +1000] "POST //connect/resources/model.php HTTP/1.1" 200 65 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.7.6)"
The IP addresses above are always changing and are from around the world. The only consistent thing is it always reports the same browser agent string.
I have changed the account passwords, removed unrecognized accounts, removed the file repeatedly but eventually gets recreated. I've also cleaned a lot of PHP files that had code injected into them.
I've tried to set up auditd to monitor the directory to see if it would reveal what is being used to create/upload the file as the access-logs are not sufficient. But any searches with that fail to show anything.
auditd rules:
Code:
# This file contains the auditctl rules that are loaded
# whenever the audit daemon is started via the initscripts.
# The rules are simply the parameters that would be passed
# to auditctl.
# First rule - delete all
-D
# Increase the buffers to survive stress events.
# Make this bigger for busy systems
-b 320
# Feel free to add below this line. See auditctl man page
# monitor unlink() and rmdir() system calls.
-a exit,always -S unlink -S rmdir
# monitor open() system call by Linux UID 1001.
-a exit,always -S open -F loginuid=1001
# monitor write-access and change in file properties (read/write/execute) of the following files.
-w /etc/group -p wa
-w /etc/passwd -p wa
-w /etc/shadow -p wa
-w /etc/sudoers -p wa
# monitor read-access of the following directory.
-w /home/members/public_html -p wa
# lock the audit configuration to prevent any modification of this file.
-e 2
Based on '-w /home/members/public_html -p wa', I should be seeing any changes, but searching with ausearch shows nothing.
Every one of these is an entry point for the bots... so long as there is a single one of them on your server you will never end the attack. It is very likely that the other file they are hitting (model.php) is being created by a hit to one of these.
it is ALSO likely that there are other vectors, so removing these will not likely end the attack. There would have been an original vector but it may have been days, weeks or months ago and was also likely removed after the initial compromise.
You cannot find "the" thing that is recreating that file because there is no single entry vector. You can only ever find "a" vector, but finding them all can be exceedingly difficult.
My Advice:
Redirect ALL traffic to a temporary "We are experiencing difficulty..." page and STOP responding to all other HTTP requests.
Change EVERY passowrd, delete EVERY FTP/SSH account and create new ones later.
Unless you are prepared to do a 100% verification of all files or if you have KNOWN good backups, delete EVERYTHING from the server and rebuild from known good source. DO NOT restore from a backup generated by or stored on the server - it almost certainly contains one or more vectors, even backups from months ago are likely compromised.
Aye, I've already been at this for several months now. I've cleaned out all the base64_decode lines that I've located with grep (most of the file dates are from back in May, so have been there for a while). Passwords have been changed and FTP is disabled. SSH is limited to one account and restricted with private keys I've generated.
I've setup crons for various find/grep cmds that will notify me as soon as a file changes or when certain things occur in the access logs, which might give me a bit more information.
I am also building a new host to migrate the websites to, so Im hoping I can catch everything before I shift it. It will definitely ensure that it uses brand new accounts, highly secure passwords, etc.
By the way, "cleaning' PHP is the wrong approach as it doesn't solve the problem. Changing passwords will work in situations where for example the client side is compromised but it won't work if the problem is on the server itself. So please look for deprecated, unsupported, stale, unmaintained versions of software you run in your web stack: any CMS, forum, shopping cart, thumbnail or gallery, themes, plugins, extensions and stuff like that.
unSpawn, I realize cleaning the php files will not get rid of the problem. Im guessing one actual exploit is being used to inject code to various files, then the attacker can use any one of the infected files to create the file used to send the spam.
However, if the code is injected again, I might see in the access logs what files are being used for the exploit. I have performed a lot of cleaning up of unused sites/files/etc, so it just going to be a process of elimination and waiting/watching.
I did find a very dated copy of Joomla on the server, which had a com_jce exploit in it (mentioned in the webhostingtalk forum link) so I've deleted that as well.
Users who aren't educated properly (or don't give a rats ass about things) will often "just" install a CMS and be done with it. On top of that there's a a clear choice to run certain themes, plugins, extensions and whatnot without bothering to check if it can be upgraded later on. So there's a lot of stalling in anticipation of b0rkage. However modern CMS versions have autoupdate functionality and will alert you via email of updates. So while it in some cases may take a nearly unjustifiable amount of effort to get things up to date there are good reasons to do so (or ditch the whole setup). Since you are building a new host to migrate the websites to the "right" approach would be to closely examine software, configurations and access controls you run in your web stack that is properly maintained and not vulnerable, migrate only those and ensure you (or the customer) are alerted of updates. An ounce of prevention, if you dig what I'm saying.
*Please be aware that, judging by the quality of their contributions, each and every board has certain trustworthy and less valuable members. Especially those who don't seem to possess practical admin knowledge should not advise others about (or rely themselves on) web-based management panels, convoluted firewall interfaces and other software suites often advertised solely with three - or four letter acronyms.
Last edited by unSpawn; 12-04-2014 at 12:50 AM.
Reason: //flow
I am not an administrator or an experienced linux or unix user for that matter. Here is an article that may be able to help you after you clean your system up. Don't forget to look at some of the links on the bottom of the page too. Although this article does not pertain to centos, you may be able to deploy this idea. Not sure if you can use this. Just trying to help. I hope you stop the bastards who are responsible.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
