Hi

Our Apache2 server (2.4.16 MPM-prefork) is sporadically entering a state where all connections are taken up showing "W - sending reply" in the server status monitor.

The server is running on a Xen VM with three dedicated CPUs and 14GB of memory.

I understand that the above issue can be caused by multiple root causes, notably poor scripts or a slow MySql connection.

I have spent days optimizing and debugging using strace, etc.

Finally, it seems that the issue isolates to degraded firewall performance. A firewall restart restores performance until degradation occurs again in the future.

Our firewall is running Debian 3.14-2-rt-686-pae with iptables 1.4.21

It performs filtering and NAT for our network. There are 8 uusers on the network with limited internet usage.

The firewall runs on a dedicated server with four Intel Xeon CPUs (3.06GHz) and 4GB of memory. The firewall also runs Squid/Icap proxy server and Snort.

Any ideas?

This is a very open ended question. It is kind of hard to say what part of your firewall might be causing issues without know how or what rules you are implying on the system. but I'll take a stab at it.

1. What logging are you doing and how much?
2. At what level are you logging if logging is turned on?
3. Are you running a stateful or stateless firewall?
4. How many open tcp connection do you have as failure?
5. Are there many rules making your firewall very complicated?
6. Are you using ipset in conjunction with iptables?

Hi Lazydog

Many thanks for your ideas - I traced some of those through.

Eventually I tracked the problem to a bug in Wordpress that was driving massive creation of wp_crons on every connection. So not networking after all.

Anywaay, with every other thing optimised over the last three weeks, our site is now very speedy!