Restart a user's server, or mass-kill processes ?
Despite having a dedicated server running my websites, I'm far from being a professional, I'm coming here for a help request, if you can, please :)
That's a long story, because, well, honestly, I'd rather write too much than not enough.
My dedicated server runs Debian Squeeze, with, installed on top of it, Webmin + Virtualmin. Each user has a different website and a different set of files in /home/username/, each website is a different virtual account.
The server was installed and configured for me by a professional, who charged me for this, that person is already working for a few of my friends and their businesses, so I know I can trust him. I can ask him for further help on the condition that I don't ask for it often, however, as much as I can, I try to fix problems by myself, that's the best way to learn.
As for me, I'm not new to Linux and the shell, but I still have a lot to learn, haha.
OK, my problem :
- one of the user accounts got compromised (my wife uploaded an old wordpress theme with a compromised timthumb.php file, I saw the problem months later when the server became slow),
- crapware was injected (my wife's blog's theme was updated with a code injection, "control tower files" in php and perl were added to /home/username/public_html/ , to /home/username/public_html/secondary-blog/ , to /tmp/ and /dev/shm (and maybe elsewhere, but then I don't know, I scanned every subfolder of the user accounts, I know tmp and dev/shm can host user's files, but I don't know of any other location)
- I think I cleaned up everything that could be cleaned up, removing all presense or reference to the .php and .pl badware, removing every goddamn timthumb.php file, using a binary comparison tool against an old uncompromised backup to check that all my wife's blog files were legit.
- however, my server is still not back to normal, some resources aren't back to how they used to be
Some elements show a recovery once I made my cleanup, fortunately, like the CPU usage returning to normal (cf the end of the "CPU day" graph http://imgur.com/d0CEk8V , as opposed to week http://imgur.com/jXeUt1i , and month http://imgur.com/I925I9v ).
But some other elements that didn't change are :
- the number of running processes, surged from around 450-500 to now around 2200
- Some elements seen in my monitoring didn't return to normal, as in these screenshots :
--> That means not everything is safe, there may be unwanted processes still running in memory, and I don't know if they'll ever stop by themselves, or, worse, if they won't reinstate harmful files on the disk :(
And I don't know how to get rid of these unwanted processes.
So... I'm wondering...
Do you know if it is possible to make the server for the compromised user restart, and only for that user ?
My hope is that only the legit processes would run this time, since the compromission files were deleted (hopefully !)
Otherwise, I also considered force-killing all idle processes, but
- I don't know how to do that ;)
- I learned this wasn't wise, since several legitimate core processes lay sleeping most of the time, and killing them could compromise the website behind the user account
In virtualmin, I clicked to restart every essential system service, one by one, as in this screenshot,
But that didn't fix the problem.
Apart from that, save a whole reboot of the whole server, I don't know what I can do... And I'd rather have to reboot the whole server just because of one user account.
Please, would you have a suggestion about it ?
Sorry for the very long thread, and sorry if it sounded confused !
- List exactly what software, themes and plugins all web sites run. Check if it is up to date, not messed with and if there's no more odd uploaded files.
- There's a scanner that detects lots of malware called "Linux Malware Detect" or LMD for short. I suggest you install and run it regularly.
evidence of HTTP encoding or system command usage, uncommon requests, seemingly unrelated error output, odd user logins, etc).
- The machine runs Suhosin, OK, but does it also adhere to what this suggests: http://codex.wordpress.org/Hardening_WordPress ?
- What access is allowed? Is anonymous FTP possible? Does SSH rely on passwords instead of pubkey auth? What other services are accessible?
- Who has an account? Are there any changes? New database users?
- Run Logwatch on your system and daemon log files as it's the quickest way to generate a list of potential issues to investigate. Check the report for "odd" requests like
- List all network connections and check for outbound requests to FTP, SSH, SMTP, HTTP and IRC ports.
- List all processes in detail and check for seemingly innocuous names. For example if your web server is "/usr/sbin/httpd" then seeing "/usr/local/bin/httpd -DSSL" or "/tmp/.ICE-unix/httpd" processes are suspicious.
* Add any details you think are related. When you reply please reply in detail (preferably in [CODE]vBB code tags[/CODE]) and attach large texts to your reply.
Moved: This thread is more suitable in Linux - Security and has been moved accordingly to help your thread/question get the exposure it deserves.
|All times are GMT -5. The time now is 10:18 PM.|