I have an aging Red Hat 9 server running on a Compaq Proliant 6500. It has four Pentium III 500Mhz CPUs and 4GB RAM. It has been running well for 5 plus years until recently. In the last several weeks it stops responding, the GUI doesn't update and you can't connect via SSH, SMTP or any protocol. Sometimes it would resolve itself, other times I'd have to hit CTRL-ALT-DEL and pray it would restart gracefully. Sometimes even that wouldn't get the machine's attention requiring a cold start. During one episode I had Ethereal running and it recorded a bridging storm, thousands of packets per second. Top showed the HTTPD process to be consuming all the CPU. If I unloaded Apache the system most times recovered. Earlier this week more problems. This time the server was having an exchange with Mozilla, in 1 hour it exchanged over a gigabyte worth of data. I blocked the IP address using IPTables and the utilization problem went away. Unblocking the entry in IPTables the conversation starts up randomly again moving huge amounts of data and bringing the system to it's knees. I couldn't tell whether my server initiated the dialog or whether the remote system did. It is always with the same IP address, 63.245.209.77 which is in Mozilla's block of IP addresses.

The machine has been fine as long as IPTables doesn't allow conversations with this IP address, I'm curious whether anyone else has had problems like this.

Lots, sadly. You're either the victim of a DDOS attack or a spammer is trying to use your machine, or someone is trying to hack you. Blocking the address is the best way to go, but I'd also strongly suggest you look at your firewalls. However these people are getting through your firewall/DMZ, into your system needs to be figured out, and they need to be blocked at the outside firewall, not at the machine itself.

I'd also be checking ALL of your other equipment for that address, along with checking for viruses/rootkits/etc....

The problem is the traffic was HTTP based. I can not block port 80 at the external firewall.

Hi there! I am having similar issue. I am new to Linux. Just wonder what kind of tool did you use to track down which IP address was the suspect (besides looking at access.log)?

Thanks for the info in advance.

When the problem ensued it was difficult to do anything. I run a home grown SNMP management tool every minute via cron. I updated that script to create a lock file and subsequent attempts check to see if prior instances of the script may still be running. If three consecutive minutes of the lock file existing I restart Apache which would restore usability to the server as all resources on the quad Pentium server had been acquired by the httpd processes. Having interrupted any activity by restarting I had some control over the server. I launched Ethereal (now Wireshark) to watch for any packets on port 80. Initially nothing but after a short while thousands of packets per second all interacting with the same IP address. I blocked that IP address in iptables and the problem did not reappear. After a week I removed that IP address from IP tables and had Ethereal watching for any conversations with it. After 270 hours Ethereal did not see any packets with the remote IP address. The problem has almost disappeared though yesterday I had three episodes where the server had high utilization. I did not catch what was the cause.

Thanks jjrowan! The information you provided looks very helpful!! I will take a look at wireshark! In my case, restarting apache didn't seem to help. I had to reboot the server every time it was hit. Otherwise, apache was not able to serve any requests from our users. I hope I can pin-point the IP address(es) that are hitting the server and block them like you did. Will keep you updated.

Do you mean Fedora 9? If this is actually Red Hat 9, it is beyond ancient and obsolete and you need to plan on getting off of this as soon as you can.

As TBone suggested, you need to start figuring out if you are simply the victim of a ddos attack or if your system has actually been compromised. I would STRONGLY suggest reading the CERT Checklist and start developing a picture of what may be going on. Simply blocking IP addresses may do absolutely nothing if the intruders have compromised the machine.

It was not a typo, it is running an ancient Red Hat 9. The machine has not been compromised. The problem has gone away having blocked the IP address strangely on Mozilla's network. They never responded to my denial of service complaint.
I'm locked into Red Hat 9 on this server as I run an application I have not been able to get running on CentOS 5.x. That application has upgraded it's code to be compatible with CentOS but unfortunately along the update path they made significant changes which disabled the functions I use most. I am contemplating installing a CentOS 4 server to see if I can get the application running on that to at least get off Red Hat 9.

Just a thought, but if you're using the packaged version of Apache that came with RH9, it's possible that there's an old exploit that someone is using to execute commands on your server? I hardly consider myself an expert on Apache exploits, but I've seen similar things before.
Never underestimate the persistence of bored script kiddies.