Apache exploit leading to arbitrary execution. Fixes?
Linux - SecurityThis forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Apache exploit leading to arbitrary execution. Fixes?
Our outward-facing http page server (running Apache) is being exploited, and I need to know how I can fix it, if anyone has a moment.
For a number of reasons, we run RedHat 7.2 on this server for the time being. All the packages have been upgraded to the latest provided by the Fedora Legacy Project. Also, the http server has no access to the truly internal parts of our network, so we shouldn't have much of a risk there.
Anyway, here is what's in the Apache error log. The "bad stuff" seems to start at line 2, and end on the second-to-last line. Obviously it jumps out at you; I just don't know how to go about fixing it:
Code:
[Tue Sep 21 04:41:42 2004] [error] [client 206.183.1.74] File does not exist: /var/www/html/robots.txt
[Tue Sep 21 06:00:10 2004] [error] [client 64.141.107.xxx] Invalid URI in request ̽&� !��!f�2sb�!*@�G!p
Process 2407 already running
rpc_server 2407 already running lets kill and restart it
Process 2408 already running
rpc_server 2408 already running lets kill and restart it
start child 2684
rpc_server 2683 started by 2655
AceShutdown try to kill process 2683
[Tue Sep 21 06:06:55 2004] [warn] module mod_rsawebagent.c is already added, skipping
[Tue Sep 21 06:06:55 2004] [crit] (98)Address already in use: make_sock: could not bind to port 80
signal 15 received
start child 2703
rpc_server 2702 started by 2688
AceShutdown try to kill process 2702
[Tue Sep 21 06:07:37 2004] [warn] module mod_rsawebagent.c is already added, skipping
[Tue Sep 21 06:07:37 2004] [warn] pid file /var/run/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
signal 15 received
rpc_server 2719 started by 2704
start child 2725
[Tue Sep 21 06:07:37 2004] [notice] Apache/1.3.27 (Unix) (Red-Hat/Linux) mod_rsawebagent/5.2.0[111] configured -- resuming normal operations
[Tue Sep 21 06:07:37 2004] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Sep 21 06:07:37 2004] [notice] Accept mutex: sysvsem (Default: sysvsem)
[Tue Sep 21 06:09:03 2004] [error] [client 64.141.107.xxx] Invalid method in request :\x81\xcc\xa5/\xa9\xc4\xccP\xde\xbd|M%\x1c\xd6S:(\x9d\x07\x0e:\x81\xcc\xa5/\xa9\xc4\xccP\xde\xbd|M%\x1c\xd6S:(\x9d\x07\x0e:\x81\xcc\xa5/\xa9\xc4\xccP\xde\xbd|M%\x1c\xd6S:(\x9d\x07\x0e
signal 15 received
rpc_server 811 started by 797
start child 812
AceShutdown try to kill process 811
signal 15 received
[Tue Sep 21 10:11:11 2004] [warn] module mod_rsawebagent.c is already added, skipping
Note: 10:11 is the time I rebooted the machine.
Also note: 64.141.107.xxx is the originating IP of the person doing this. I've already tracked down who it is, and at this point I just want my machine in a reliable state.
The httpd process is being exploited and becomes some sort of rogue such that it can not be identified as controlling the network connection that the malware opens, when I run
Code:
netstat -nep
The only way to get things "back to normal" (to get httpd up and running again and kill the malware) is to reboot.
Distribution: OpenBSD 4.6, OS X 10.6.2, CentOS 4 & 5
Posts: 3,660
Rep:
Hmmm, are you sure the attacker is actually able to execute commands? From the logs it seems like it's just a DoS
that forces Apache to hang or restart (06:00:10). The later request (06:09:03) is attempted shellcode, probably as
part of an attempted buffer overflow. It's not apparent that it was successful, although it may have been.
Does your log usually have messages about rsawebagent already being installed? If it does not, than it would appear
that the attacker is trying to force it to load, but it's already loaded. This would seem to indicate that it's the
rsawebagent that is vulnerable and the attacker wants to make sure it's loaded even after he's exploited the system
so he has a way to get back in if he needs to (i.e. by exploiting that module again). If that module is software
purchased from RSA Security, I would contact them immediately.
Yeah, the program that runs looks to be an IRC bot. It connects to an irc server on port 6667 (as seen in netstat -nep, but remember that the PID is not displayed, which tells me that it's a process that has been munged), and sends data back and forth, which I have observed with tcpdump.
I've actually been wondering if it isn't an httpd exploit, but something that http loads/runs, like you say. We do run RSA SecurID on that webserver. Normally it is encountered by going to the site on https, accepting the (OpenSSL-generated) certificate, and entering the SecurID. I'm going to look for more vulnerabilities there.
Originally posted by chort The later request (06:09:03) is attempted shellcode, probably as
part of an attempted buffer overflow. It's not apparent that it was successful, although it may have been.
Well, if you look at the logs, it gives the error (at 06:09:03) about the "invalid method," and then there are no more log entries until the SIGTERM (signal 15). I'm not sure if that SIGTERM is from my reboot or not, but I do know that once this exploit is taken advantage of, httpd doesn't respond to requests, and isn't listed in ps -A.
Edit: Oh, but by the way: when I tracked this guy down, the IRC bot that runs on my machine has an info tag that says "Apache Scaner"[sic].
Update: /var/log/httpd/ssl_request_log has no entries for today until I tested it half an hour ago. In fact, I do not seem to have any entries for any early morning hours (this hijack always occurs around 7:40 AM local time), which may or may not be interesting.
Distribution: OpenBSD 4.6, OS X 10.6.2, CentOS 4 & 5
Posts: 3,660
Rep:
So it's very likely a mod_ssl exploit, if the attacker is scanning for other Apache installations.
What version of mod_ssl are you running and what version of OpenSSL is it built against?
Or of course, it could still be the rsawebagent and it's only vulnerable when running in Apache.
I'd still contact RSA support immediately to see if they've had other similar reports.
After looking through these entries, I'm feeling ravenous. Below I'll ponder what's happening:
64.141.107.xxx is the address of the machine (possibly exploited itself; usually this exploit is executed by a hijacked machine in a city in the U.S., or another machine in Australia) in Canada that initiates the connection and executes the exploit. I'm not sure why there are two separate exploit attempts nine minutes apart (and bookending another request). Notice, though, that each exploit attempt results in a different return code (400 and 501, respectively).
217.10.216.xxx tracerouted to an ISP in Romania. Romania is where I suspect very strongly the attacker is based. Could this be his home IP? Maybe. Maybe he is checking on the progress of his exploit. Maybe he is checking out the computer he's exploiting. Maybe there is another script running that somehow communicates to a program that is in my machine as a result of the first exploit (the 400) actually being successful? Look how closely the requests in the middle bookend those from the machine in Canada. Is there anything significant about the request to the client_restricted/secure.html page (the RSA SecurID authentication page) twenty seconds after the request to the main page? I'm guessing so: I'm guessing he's actually using a browser. Notice also that the request seems to be directed right at my machine's IP ($LEASED_IP).
Can anyone tie all these clues together to tell me "this is the exploit that's happening"? Also, what are your feelings on how confident I should be that the 217.10.216.xxx IP is the actual person responsible?
This is the log from yesterday. Our server was indeed hijacked yesterday, as the log hints at. From this, I think I know now exactly what was happening today, as reflected in the log:[list=1][*]On 2004-09-20, our server is exploited by a machine designated by the IP 139.130.163.xxx, which is a co-opted Australian computer.[*]On 2004-09-21, the same exploit is run. Only, it doesn't work: his bot hasn't logged into IRC. The attacker opens his browser and loads our site (by IP, remember) to check whether the web server is running. It is. Interestingly, he follows a link to an https location of our site that is a login screen expecting an RSA SecurID. At this point he:[*]Runs another exploit on the same machine. This time it is successful.[/list=1]
I am confident at this point that the attacker either screwed up, underestimated his victims, or is just plain ignorant of the mistake he made when he loaded our site from his home computer. Unfortunately he is in Romania, so I don't know what can be done. Surely the FBI can do something.
Distribution: OpenBSD 4.6, OS X 10.6.2, CentOS 4 & 5
Posts: 3,660
Rep:
Offload all the logs, md5 and sha1 each of them, and burn them to CD along with their respective checksums,
then seal the CD in a container and place your initials and the date on the outside of the container. Perferably
you should use some type of tamper-evident seal on the container. This is evidence should you decide to
pursue a criminal investigation. Better yet, make bit-pefect copies of the HDDs in that machine following
the same steps above.
You could shed some further light on the subject by listing all the modules which Apache is loading. That will
help to define the field of possibilities a little better about what could be the source of the vulnerability. Just as
a wild guess, it seems to match this vulnerability
fairly well. You could also try reviewing other Apache buffer overflows.
Barring a 0day (which is doubtful), I'd say you're a few Apache updates behind. The best possible thing to do
at this point would be to migrate to a more recent and more well-supported (with security updates) OS.
We haven't upgraded anything because the anacondas for RH 8.0 and RH 9 crash reliably. I've been planning on upgrading Apache by source, but for now we have the latest provided by the Fedora Legacy Project (2.4.20?). Also, I wanted to keep gathering information.
Thanks for the link to that site, it looks very helpful! And thanks for the advice!
Distribution: OpenBSD 4.6, OS X 10.6.2, CentOS 4 & 5
Posts: 3,660
Rep:
I wouildn't necessarily go to Apache 2.x just yet. Personally, I find nothing wrong with 1.3.x and if it
were me deploying a server in production, I'd stick to it. I'm a little confused though, because
the version number you gave could also be a kernel version. I would rebuild the server rather
than upgrade it, because now that it's been compromised you don't know what else might have
been tampered with.
Oh whoops, yea that was the kernel version (we're running 2.4.20-28.7). The version of Apache we're rockin is 1.3.27, which I suppose is a little old, so I figure I'll build 1.3.31 from source today.
Yeah I have no problems with 1.3, and more importantly I haven't heard any compelling reasons to upgrade.
Thanks for the tips and the feedback -- I know mods are often underappreciated. I'm much further along than I was two days ago!
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
