Hello all,

Right now a user has exploited apache (apparently) and is running an IRC connection on my computer as a result. This happened yesterday, so I upgraded Apache and investigated. Luckily I was running tcpdump all night because I was suspicious already, however tcpdump wasn't dumping packets so I couldn't get any juicy IRC bot login information. Last night we started intercepting raw packets, but I can't get to them just yet.

Anyway, the user is on my system now, and I'm pretty sure I have the guy on IRC responsible for this nailed. The initial connection to my computer resolves (traceroutes) to telstra, an Australian ISP, which is not the same location as the person I think is responsible but I guess that's common (let me know your thoughts?).

So my question is: does anyone have any tips on what I should do? I haven't been in this position before, and now that I have this guy red-handed, I want to know what I should and can do.

Thanks!

If it were me, I'd kick him out, patch tyour security holes, and don't play detective.

rjtucke

Well, normally I might agree with you, but we've had this server go down on us several times in just the past few weeks. In the first two or three cases, the binaries were all corrupted and I had to do a backup, format, reinstall and reconfigure. This has cost us at least a day in each case. If the same person has been responsible for any of these failures, either directly or by "drawing fire" to our IP by his actions, I want to defer it to the proper authorities. I've already done most of the legwork in that I have the person's IP which can easily be associated with their address by their ISP.

If it was just a script kiddie or a worm wasting our time, I might not care so much, but this is a deliberate action.

Oh my. Well please accept my condolences. I am simply not qualified to advise how to trap a cracker. Good luck.

rjtucke

Sorry for what happened, these people are wasting so much time for making our lives so complicated! I can't undestand them.
Anyway, if you look in the past threads, you can easly find some tips by unSpawn and Chort.
BTW, I'm missing something about this case description: are you telling us the box has been hacked several times or it just crashed several times?

I think there's a disconnect between the people who run the magical software and their realization of what the software really does. I mean, all he's doing is running a bot to maintain his ops on IRC, and the result is a cost to us of a lot of time, money and reputation.

Anyway, yeah the first time the server got corrupted, I was thinking maybe it was a fluke. The second time, I suspected hardware (even though the server was using RAID 5), but the third time (and even the second time) I suspected a hack. Now that I've caught this guy twice in as many days, I'm thinking that he found our server and has added it to his "favorites." If only I could be sure this was the same person that caused the system failures! Our old server is still sitting up there, so perhaps I should check the error logs again, however I don't remember seeing anything interesting before. The only interesting thing I noticed the first times was that the system had been rebooted at strange times, mostly during the weekend. Usually init and several other core process images had been corrupted, and their timestamps modified. That's what bothered me: their timestamps were modified. Also, the permissions for init were modified such that the user's group was given write access.

Checking the Apache error log, the last entry is a message about a malformed protocol string. So I believe my suspicions about a buffer overflow exploit in apache are probably correct, especially since the initial connection is made through port 80.

Thank you for the pointers!

For security reasons a day of reconfiguring - is worthy!!! Another action u should do, is identify what proccess is the hacker using... something u dont use for sure - and remove the damn package!!!

But above all are u v sure he's already inside? Crashin many times is quite a problem... but knowing that he is inside... its kinda strange? Through apache i guess he can only start/stop services... but setting them up i think not...

At least install www.snort.org to get rid of the most and various attacks - after u are sure u'r clean...

3spre10:

Here's what seems to be happening:
- He connects on port 80.
- He sends some crazy string which allows him to execute arbitrary code. He starts his program in my server's memory.
- Apache becomes defunct (netstat -nep shows no PID for the hijacked process).
- He disconnects. The program keeps running.

The original connection on port 80 traceroutes to Australia (telstra.net), but I'm assuming this is a bounce. I'm 90% sure I know the user who is doing it and what his real connection is (and where it is).

In my opinion then - u should follow these steps :
- download snort
- stop webserver service
- unplug external cable for some time (exactly the time that would take to install and configure snort - so print your documentation before or just memorize it - its not too much to remember)
- start snort and be sure it is running
- plugin the external cable
- update webserver
- start webserver and check if snort is running.

Prepare these steps before - anyway u should check with rkhunter where could be the potential faults off access.

If you are using a webserver with more than 1 host u should consider on planting a "decoy" in http://your.ip as the 1st virtualhost in a different folder with its own cgi-bin and stuff. This is a trick that helped me a lot!

That's a way.
Another thing you can do, and this is nasty, is building a honeypot for trapping this guy and collect all the information you can about it. It's a lot of work though.
The modus operandi is quite strange, it hardly can be and apache bug so i presume it's a code injection is some dynamically generated web page.

I would love to get all the information about this guy that I can. I have been watching him and tracing him for a couple days now. I know of another box that he has hacked: a DSL subscriber line that runs an http server. So, he's using this same hack to take advantage of another box, only this guy isn't savvy.

I plan to leave the packet grabber running all night so I can get his bot's login information. From that point, assuming his password is generic, I should be able to get more information about him.

What I'd like to do is contact this other guy that has been hacked, and his ISP, and somehow get this loser in trouble. Is that feasible?

My best advice:
don't involve yourself in trying to trace this guy. Best case scenario, you get some lame script kiddie from causing a minor hiccup in the internet. At worst case, you piss off someone who knows just enough to cause a lot of headache (my friend went down this road, and ended up being the subject of a huge DDoS that brought down core routers in the surrounding areas).

Take some time, and build a CHROOT jail on your next attempt, and use something like grsecurity in addition to snort, and a stack-guarding compiler. Really configure your apache package inside and out. And forget about this guy.

Aaron

Couple of questions:
1) What version of Apache are you running
2) Could you post the malicious http request string? (If you're uncomfortable posting it (if you think it's a 0day or something), you can email it to me instead.

Some advice:
You should isolate that box from the rest of the network, whether physically or by making it into a DMZ with your firewall (basically get it away from another other machines). Second consider all passwords and authentication tokens to be compromised as well. As far as reporting, you should make an image of the drive if possible, but definitely send any relevant log info to your ISP as well as the crackers. You can usually find an abuse@ contact with a whois query. If your machine is being used to attack other systems, you should take it offline untill you can be sure it won't immediately get re-hacked.

As for prevention, chroot and grsecurity are good ideas (as mentioned), including some kind of stack protection like PaX or Exec Shield. Might want to consider mod_security as well. There is also some good reading on general Apache hardening here:

http://www.securityfocus.com/infocus/1694

If this guy is really persistant, you could even do some kind of simple string filtering with iptables or application level filtering with Zorp.

The very first thing you should do is create a bit-perfect copy of the local storage media (i.e. hard disk drives). You should rebuild the machine from scratch. Zero out the disks (i.e. write zeros to every sector) and reformat them. Install the OS, and before plugging it into the network, use another machine to download all the security updates and put them on some type of portable media (CD-R, USB Flash, etc). Install all the security updates from the known-good media and reboot prior to plugging into the network. You should also seriously consider implementing some stack protection, such as PaX (like Capt_ suggested) and a Host Intrusion Detection System (such as AIDE, or Tripwire).

If you do plan on keeping the storage media as evidence, make another copy of them (you'll need one for evidence and one for yourself). Make sure you generate checksums of the file system (using two different hashing algorithms is recommended, for example md5 and sha1). It's also important that you seal the media in some type of container and put a label on the outside with the initals of whomever made the copy and the date it was made. It's a good idea to put some type of tamper evident seal on the container as well. All of that will help to ensure that you evidence is acceptable for a criminal case.

The most important thing you should do is try to correlate traffic from the compromised machine to see if it's been used to attack any of your other machines. The IRC bot might not be as simple as just holding Ops on a channel; chances are very strong that your machine is now being used as a Zombie in a Botnet, i.e. it can be called on at any moment to participate in a DDoS attack, send spam, start propagating a new worm or virus, etc...

You should definitely check your firewall and/or router logs that would show traffic between this machine and other machines on your network. You should also check the logs on your other machines to make sure the compromised machine hasn't tried to contact them. You can also put the hard drive with the disk image into a workstation and mount it read-only to examine the logs thoroughly to help determine if it attacked other hosts.

Remember, nailing this attacker isn't as important as protecting your network and making sure that it's not further exploited. Worry about securing your systems first, then go back and look through the evidence that you've collected. Also, like Capt_ said you should change the passwords on all the rest of your machines that might have had the same passwords as any on the compromised machine. If you really want to do it right, you should change all your passwords regardless of whether there was an overlap. If there were any private keys stored on the compromised machine, make sure you remove the corresponding public keys from authorized_hosts on all your other machines.

I'm very interested in patching my kernel with PaX. Honestly it escapes me why the default kernel allows things which it knows to reside in non-executable space, to execute. When a program is loaded, it is known where the code rests, so surely it should be fairly straightforward to deny executable access to non-code areas, and it should also be easy to control write access to code areas.

But, does PaX have any limitations or drawbacks? For example, is it slow? Does it not prevent all of these types of exploits?