Hi,

I have a webserver (Debian 8.3 with Apache 2.4 and PHP 5.6.17) and the whole server seems to be infected. I have shut it down for now to investigate and I'm going to setup a new server. BUT, I don't want to just transfer everyone's html_doc to the new server as there could be infected code in there.

There seem to be other processes running with weird names such as "././crond" and "/usr/sbin/apach" running by a sites user.

I've tried various scanners, example 'maldet' and various rootkit scanners etc.
Maldet has detected some code but I can still find files with malicious code in when I look in them.

My question is, how do you guys handle this kind of situation?
Are there other tools to find more dangerous code? And how do you find out HOW they manage to get in to the server in the first place?

I'm very interested in this as I don't want to find my new server infected again and start all over.

I have around 30 sites on this server.

Thanks,
-Patric

First and foremost: https://www.linuxquestions.org/quest...erences-45261/
Not that I'm typical, or even good at this, but I usually start with the logs in /var/log/* and I look for what stands out.
"sites user" seems like a good starting point.

I usually check for incorrect file permissions.
Anything not File=644 and Directory=755 is immediately suspect, unless it's expected, such as .cgi stuff.
and then compare those files from suspicious stuff in /var/log/*

I do not allow the customer to dictate what permissions they "need".
If you allow "users" to ftp in and edit /var/www/some/path, then ftp credentials are immediately suspect.

I check the logs for files being dropped in /tmp or /var/tmp
I check the logs for wget, curl, lwp-<something>, exec, rm -f, unzip, tar...

using maldet 1.5
clamscan will tell you what's in files that it says are infected.
clamscan does not clean anything, it can move or delete infections, if told to do so, and easily identifies infections.

With 30 some sites, I'd
which will scan recursively and report only infected files.
Examine /root/scan.rpt closely and compare to hits in examining /var/log/*

Wordpress or other popular target platforms? e107, Joomla! up-to-date?
Not up-to-date Plugins are ruthlessly targeted.
What Content "system" are these sites using? What is the common-denominator?
All of them up-to-date?

apache should run as httpd/www-data

That's a basic start. Other more experienced board members, and security pros will offer other techniques I may have missed
without my daily caffeine.

Man, this was not fun. I scanned the entire server with clamscan and found a bunch, I then examined the logs to see what files where accessed etc.
I thought I got it pretty much cleaned up so I booted it up. It was pretty quiet for a while and then I found that the permissions where all wrong on each vhosts html_doc folder so you could technically access anyone's files if you wanted. I fixed this and then it pretty much stopped.
There where some files left but I think I have found the majority of them now as I've been watching the logs.

I also found a "bug" in maldet. lets say you are in the folder /var/www and run "maldet -a example.com.folder" it won't scan everything in there. For some reason you have to specify the whole path, in this case /var/www/example.com.folder and then in started to scan everything, with found some more stuff.

Thanks for the tip and everything. This was not fun at all!

Glad it worked out.
I always specify the full path using maldet.
Not a bug. see Keep an eye on things.
I use this on a critical server (that can send emails)
Save as maldet_scan.sh somewhere like /root/
The above script will run and email you the last report from that scan.

Change /opt/zimbra/store to /var/www/example.com.folder
and you should be golden.

Maldet will run every day via cron and email you the report from the run (server permitted) at user@domain.com
Good Luck.