Hello,

Despite having a dedicated server running my websites, I'm far from being a professional, I'm coming here for a help request, if you can, please

That's a long story, because, well, honestly, I'd rather write too much than not enough.

My dedicated server runs Debian Squeeze, with, installed on top of it, Webmin + Virtualmin. Each user has a different website and a different set of files in /home/username/, each website is a different virtual account.
The server was installed and configured for me by a professional, who charged me for this, that person is already working for a few of my friends and their businesses, so I know I can trust him. I can ask him for further help on the condition that I don't ask for it often, however, as much as I can, I try to fix problems by myself, that's the best way to learn.
As for me, I'm not new to Linux and the shell, but I still have a lot to learn, haha.

OK, my problem :
- one of the user accounts got compromised (my wife uploaded an old wordpress theme with a compromised timthumb.php file, I saw the problem months later when the server became slow),
- crapware was injected (my wife's blog's theme was updated with a code injection, "control tower files" in php and perl were added to /home/username/public_html/ , to /home/username/public_html/secondary-blog/ , to /tmp/ and /dev/shm (and maybe elsewhere, but then I don't know, I scanned every subfolder of the user accounts, I know tmp and dev/shm can host user's files, but I don't know of any other location)
- I think I cleaned up everything that could be cleaned up, removing all presense or reference to the .php and .pl badware, removing every goddamn timthumb.php file, using a binary comparison tool against an old uncompromised backup to check that all my wife's blog files were legit.
- however, my server is still not back to normal, some resources aren't back to how they used to be

Some elements show a recovery once I made my cleanup, fortunately, like the CPU usage returning to normal (cf the end of the "CPU day" graph http://imgur.com/d0CEk8V , as opposed to week http://imgur.com/jXeUt1i , and month http://imgur.com/I925I9v ).

But some other elements that didn't change are :
- the number of running processes, surged from around 450-500 to now around 2200
- Some elements seen in my monitoring didn't return to normal, as in these screenshots :
http://imgur.com/e3ljl4C
http://imgur.com/CtV90xU

--> That means not everything is safe, there may be unwanted processes still running in memory, and I don't know if they'll ever stop by themselves, or, worse, if they won't reinstate harmful files on the disk
And I don't know how to get rid of these unwanted processes.

So... I'm wondering...

Do you know if it is possible to make the server for the compromised user restart, and only for that user ?
My hope is that only the legit processes would run this time, since the compromission files were deleted (hopefully !)

Otherwise, I also considered force-killing all idle processes, but
- I don't know how to do that
- I learned this wasn't wise, since several legitimate core processes lay sleeping most of the time, and killing them could compromise the website behind the user account

In virtualmin, I clicked to restart every essential system service, one by one, as in this screenshot,
http://imgur.com/fvQUiBu
But that didn't fix the problem.

Apart from that, save a whole reboot of the whole server, I don't know what I can do... And I'd rather have to reboot the whole server just because of one user account.

Please, would you have a suggestion about it ?

Sorry for the very long thread, and sorry if it sounded confused !
Greetings,
Sabinou

...and as long as it contains the necessary facts that's definitely appreciated.


If that's the same server as in this thread then you had about a years time to learn ;-p


...which points to constant monitoring, timely alerting if conditions change, knowing what users run and educating users.


Well, you certainly did a lot and that's commendable. Removing that vulnerable software, including themes and plugins, helps avoid exposing the same problem again. Comparing against a known safe backup verifies files are (or not) as they were. That's good. The question is if you used any old backup to restore files. If the backup was taken before you realized the web site was compromised then those files (configuration files, database, .htaccess, plugins, themes, includes, uploaded files with wrong extensions) may contain the same or other vulnerabilities. And to make sure: even if you didn't restore from backup the site, or an adjacent one, may have other vulnerabilities and if you didn't change passwords then a per could well use the old credentials.


There's a few things you should do, order doesn't matter, doing them does. And please don't rely on your web-based panel but instead use software the system provides. Want to know about query performance? Run MySQL Top. Want to know about page requests? Run Apache Top. Want to know what processes run? Use 'ps'. Want to know what processes take up most of the resources? Use a SAR like Atop, Dstat, Collectl (or even play sar). Want to know what files a process keeps open? Use 'lsof'. Want to know about errors and anomalies in your system and daemon log files? Run Logwatch. Etc, etc. A web-based management panel helps those who have theoretical knowledge and practical experience but for those that don't it doesn't.

- List exactly what software, themes and plugins all web sites run. Check if it is up to date, not messed with and if there's no more odd uploaded files.
- There's a scanner that detects lots of malware called "Linux Malware Detect" or LMD for short. I suggest you install and run it regularly.
evidence of HTTP encoding or system command usage, uncommon requests, seemingly unrelated error output, odd user logins, etc).
- The machine runs Suhosin, OK, but does it also adhere to what this suggests: http://codex.wordpress.org/Hardening_WordPress ?
- What access is allowed? Is anonymous FTP possible? Does SSH rely on passwords instead of pubkey auth? What other services are accessible?
- Who has an account? Are there any changes? New database users?
- Run Logwatch on your system and daemon log files as it's the quickest way to generate a list of potential issues to investigate. Check the report for "odd" requests like
- List all network connections and check for outbound requests to FTP, SSH, SMTP, HTTP and IRC ports.
- List all processes in detail and check for seemingly innocuous names. For example if your web server is "/usr/sbin/httpd" then seeing "/usr/local/bin/httpd -DSSL" or "/tmp/.ICE-unix/httpd" processes are suspicious.
* Add any details you think are related. When you reply please reply in detail (preferably in [CODE]vBB code tags[/CODE]) and attach large texts to your reply.


[EDIT][/EDIT]