This is insane. Over the past month, my DSL has been quite busy, and I couldn't figure out why. I started looking at the router logs, and discovered that my webserver was making a lot of connections to ad sites (see 60 second log below)
Finally notice something I should seen weeks ago.
My httpd processes look odd. First off, I usually 2-4 processes running. Now I have, like, 30 of them.

Secondly, instead of a simple 'httpd -D', the process lists like this:
Stopping the httpd process does shut down all the processes and stops the traffic on my DSL, so I feel this is the source of the problem. I just can't figure out why the server is suddenly reaching out to touch, well, everyone?

I think the problem is going to be somewhere in the daemon script, or one of the .conf files, but I'm not sure what a "normal" one looks like. Any help would be appreciated.

(Now *this* would have been a good thread in the Linux - Security forum)

I just can't figure out why the server is suddenly reaching out to touch, well, everyone?
Because you didn't take care of your server, so someone else did. Since they felt comfy, they decide to stay and upload or make your server fetch some bot software to make them money. Note it doesn't need to run as root account user, so chances the box is cracked to the core would seem minimal: but do check.


I think the problem is going to be somewhere in the daemon script, or one of the .conf files, but I'm not sure what a "normal" one looks like.
The problem most likely is in running some deprecated, stale, outdated, vulnerable version of software (most likely some package using PHP) unprotected.
Let it run, make a full "ps axfwww 2>&1>/tmp/process.log" output, one netstat output: "netstat -anp 2>&1>/tmp/connections.log" and (just in case) one lsof output: "lsof -n 2>&1>/tmp/openfiles.log". Now check for processnames that seem "odd" and you most likely found your culprit.

0. Raise your firewall to DROP any inbound traffic to (running) services.
1. kill all network-facing services you don't need (httpd, any remaining accessable Perl, PHP or other interpreter processes)
2. Remove the rogue processes you found in your process, connections and openfiles logs. Most likely their files will reside in directories the user has access to like /tmp or /var/tmp.
3. Download, install and run Chkrootkit, Rootkit Hunter. Run "rpm -Va --noscripts 2>&1>/tmp/rpmcheck.log" and check output. If something seems fishy: please post *exact* output.
4. Remove any software that reads "only for development use" (like for instance XAMPP) or make sure you deny access (firewall) and only allow LAN usage. Update all packages and make sure you do this regularly from now on. Install mod_security. Make sure you deny any access to services until you have properly hardened them.

*Maybe request this thread to be moved to the Linux - Security forum.

0. Had that covered for years. Have a hardware firewall will all ports locked except 80, 25 and 110 (the whole point of the box is a web/mail server). It's configuration has not changed in two years.

1. Since it *is* a web server, I kind of need httpd running. But when I rebuilt the box in January, I did disable FTP, Telnet, SMTP, news... all the stuff I'm not going to use.

2. Killed the wayward processes two days ago. Still need to find a "normal" httpd script to compare mine to.

3. Will do it over the weekend.

4. I'll look into that. The LAN firewall is covering my whole net, so I should be good there. Can't restrict to LAN-only (that whole web/mail server thing again).


I'll work on it over the weekend and post back with results. In the meantime, if someone could post their httpd script, I'ld be eternally grateful.

1./2. what I mean is wrt ops around cleaning up, just in case something lives you overlooked. But if you don't want to make customers see downtime or whatever reasons you have for not bringing stuff down: BMG, you're the expert I guess...

Well, cleaned up the mess.
Uninstalled Apache and reinstalled. Took five minutes and the system is as good as new.

In the meantime, I'm reviewing my router settings and the server security.

I agree that running the minimum services needed is the most secure way to run. I just need to look closely at what is *really* needed.