Sorry about butchering your handle. Feel free to call me Frito Bandito in retaliation, all the kids in elementary school did.

Very true, a backup that has never been restored is no backup at all. For it to be "known good" it must have been created before the compromise AND been test restored somewhere, preferably on a warm standby, but in a QA/lab environment counts too. Practice also helps minimize downtime when the real thing happens.

Money and reputations are at stake. So while I agree in principal, I'm also a realist. Quitely handling the problem is preferable to denying there is a problem, which is probably what his bosses are telling him to do right now. Thanks to Sarbanes-Oxley, most execs don't even want you to email them about these things anymore. Just make it go away and make them look good in the process!

To the OP, do you not have a site mirror you can move onto temporarily?

If you are hosting sites, maybe you should have one.

Also looking towards running your stuff via a VM would be a good future idea. If you had been doing so this time, it would have been as simple to fix as reloading the VM image.

I'd love to know how you managed to get compromised. I've been running my own sites for years, with mail servers, ftp, etc, and never been compromised yet. Got compromised a couple of times when i first started out with linux, but you learn fairly quickly how to avoid it after it happens. Was your machine kept up to date with the latest patches, bug fixes, etc? Was your root password of a weak dictionary word type? My root password is an 18 alpha-numeric character password, using no dictionary words, just upper/lowercase letters and numbers. Were you using certs for ssh and running it on a non default port? Same goes for ftp?

I use a non httpd web server, ftp over ssl (with no non-ssl sessions allowed) on a non default port, ssh on a non default port, selinux and aide on the server, and it is updated daily. I havent seen any attacks in a long time for ftp or ssh, and only get the usual php attacks on the web server, but php isnt installed.

Just my thoughts on this.

The OS is redhat linux 9.0

Server version: Apache/2.0.40
Server built: Jan 27 2004 03:22:55
Server's Module Magic Number: 20020628:0
Architecture: 32-bit
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D HTTPD_ROOT="/etc/httpd"
-D SUEXEC_BIN="/usr/sbin/suexec"
-D DEFAULT_PIDLOG="logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="logs/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"

sshd version OpenSSH_3.5p1

>>Was your machine kept up to date with the latest patches
No, never upgraded nothing, I also never used Red hat upgrade facility...

>>Was your root password of a weak dictionary word type?
The root password was not a problem

>>Were you using certs for ssh and running it on a non default port?
yes, using certs (I think this is default) and using non standard port.

FTP is - ProFTPD Version 1.2.9 running on port 21 commonly.

-----------------------

I think they get in through apache.

I remember on Feb 11 there was an apache attack, hundreds of processes raise the uptime to 20.0.
I have a little daemon that kills "strange" processes then issues an email warning. It found this:
several:
apache - 9442 - ? - bash
then:
apache - 9793 - ? - bash
root - 9893 - ? - [sh]
apache - 9895 - ? - ./hat

More or less often I saw "root - 9893 - ? - [sh]" (with different pids, of course) but as I did not know what it was, I just ignored them, but the "apache - 9895 - ? - ./hat" was something new.

Several months ago they managed to get in trhoug a non safe_mode enabled php site, throug phpBB bug, they managed to download a file (wOot i think I remember, a binary) to the /tmp dir and executing it, somehow that process launched as apache was able to gain root privs and open a pseudo terminal and a listening port 1666 (i think I remember)... I discovered it, found the script kid ip that did it, called the ISP asking for punishment but I was completly ignored. Since that day all php websites have safe_mode ON. I think I completly removed that threat.

Now I think the problem is apache, I've made /tmp nosuid,noexec,nodev,rw and also /dev/shm in the hope this will harden things for them, but I not fill very comfortable. I should upgrade apache, but I can't because I don't know if the frontpage extension for linux will work with version 2.2 I have an important reseller that uses FP a lot.

I found this in a forum, this should be my next step (or completly rebuilding the system):

- mount $TMPDIR (usually /tmp) noexec,nosuid,nodev
- chroot apache
- run apache on port 8080 only and ipforward 80->8080 (this means no part of apache has root permissions left
- of course run the most current apache version (2.0.53 today)
- have another non-apache webserver at hand in case there are unresolvable problems.

A reinstall is your only option at this point.

Chances are the guy who has your machine owned at this point is using it for spam, maybe as an attack vector to gain access to your clients (which could make you very liable if anything serious is stolen from them or he hacks them and they find out you've known about that machine for some time), possibly as a place to store child porn or similar illegal things and offer it up to the masses. Are you getting my drift yet. It could end up with a mass of trouble being dropped on to you.

The chances are he is using a bot or hacked computer (if not multiple ones) to access your system, so dont bother with the report to isp, unless the ip is resolving to a big corporate company.

I dont know how he has rooted you, but my guess is your inability to update has made it very easy.

The proftpd version you have if less than rc3 has multiple known exploits available for it.

http://www.google.co.uk/search?q=proftpd+1.2.9+exploit

If he has a rootkit on the machine it likely has backdoors opened. To find them scan the ip with nmap and look for all 65535 ports. Anything open that you havent opened and cant find info on is likely a backdoor.

Best advice is take it down, reinstall, put it back up. It may not be ideal for you and your clients, but if anything happens to them due to you not doing anything you will definetly lose them and tarnish your name in the sort of way you dont easily recover from. Be honest and show yourself as being proactive when it hits the fan.

I posted a list of steps to follow to mitigate and gain control of the situation. By now I should be able to draw a conclusion by data that was returned. So, let's:


1. When did this happen?
Timeline unclear, no factual data presented.
- "Several months ago they managed to get in trhoug a non safe_mode enabled php site"
- "it could have been months ago, I saw couple of evidences but I was too bussy"
First known activity (YYYYMMDD) by "evidence": 2006/03/10
Last know activity by "evidence": 2007/02/20


2. How did this happen?
- no SW was updated. Statistics based on presented SW versions:
1. RHL 9.0: 35% of the flaws where moderate, 8% high and 2% critical. 60% of attacks could be done remote.
2. OpenSSH 3 series: 30% of the flaws where critical and nearly 90% remote.
3. ProFTPd 1.2 series: 50% of the flaws rated as moderate, 25% as high and nearly 90% remote.
4. Apache 2.0 series: 30% moderate, 3% high, with 80% remote.
- no regular auditing was done.
- warning signs where not followed up with proper measures.


3. Detailed account of the attack?
Taking stock of "evidence" presented: impossible.
- system authentication data: none.
- IDS, router logs, filesystem integrity checkers, package manager: none.
- all system, daemon and firewall logs: partial, inconsistent, incoherent.
- running services: partial, inconsistent and incoherent.
- unusual (setuid root) files: none
- user shell histories: none
We don't have any entry vector data so we can't set up a compromise history. Only thing we have are some dates, tools used (SHV4/SHV5 and remote vuln "ftp_scanner").


4. Detailed assessment of damage done?
Impossible, but since compromised to the core everything should be marked as contaminated.


I found this in a forum, this should be my next step
A list of steps was presented to you to follow to mitigate and gain control of the situation in case you needed guidance. You chose to ignore that but did not use a structured approach yourself to assess your problems. You presented your findings partially and in an inconsistent and incoherent way which doesn't help us help you. You also drew a conclusion based on no presented evidence at all, and that setting safe_mode and mount flags "will harden things for them".

Putting it bluntly:
- There is no way you will be able to "save" or "restore" this box and data. Deal with it!
- The problem isn't Apache: the problem is you, for gross, deliberate and continuing negligence.
- You are a risk: you lack the skills and professionalism to stay in this business. Hire an admin.

If you would have shown you would be willing to learn, absorb and apply knowledge and skills and deal with it in a way that benefits us, your clients and yourself I would invest more time to help you. But based on what you presented sofar that would be a futile investment right now.

>>Putting it bluntly:
>>- There is no way you will be able to "save" or "restore" this box and data. Deal with it!

I think I can handle for some time unless until I gather the will to rebuild the box (I'm little lazy right now). So it will keep working for now...

>>- The problem isn't Apache: the problem is you, for gross, deliberate and continuing negligence.

I did not hacked the box. I have good feelings in these situations and I sense it will be ok for some time at least, however I agree there is some danger but I have moved all my important files away from that flaky server a long time ago, only the clients remain to be adviced, and I swear they will be very soon.

>>- You are a risk: you lack the skills and professionalism to stay in this business. Hire an admin.

I cant afford a good admin, so me is what there is for now.