Hi,

I have this server which traffic suddenly increased very fast. I noticed that after sometimes the site fail to load and our server domain name in that NS (in the same machine) failed to be resolved.

Let's say the Server A is our web server and also our NS. Server B has domain theexample.com pointed to Server A as its Name Server.

When the Server A "crashing", I couldn't query theexample.com from the Google DNS. But after I restarted Apache (yes, only the Apache) in the Server A, the Google DNS could found theexample.com in Server A.

I actually have two questions :

- I thought our server was 'ddos-attacked'. I checked with this command from the machine itself
# time wget -O /dev/null localhost.

The response time normally is below 0.9s. But when the error occured, the number is 1m which might be a sign for DDOS attack.

But If I look using
# netstat -an | grep "theserverA-ip-address:80" | wc -l

I found all the active connection to http port (in any state) is around 1746. While the SYN_RECV state is only around 70. So I believe the number was still normal right? The ESTABLISHED state was around 300. TIME_WAIT 330. And in system messages there were no "possible syn flooding attack" error log.

If it was not DDOS, what else might have caused the web server acted like being hit by one? I don't found any useful errors in apachelog. There were no "too many connections in mysql" or "MaximumClients reached, considering increase it". So the apache is still running normal.

- The second one. The server has enough free resources when the problem occured. The bandwidth was only 20% used, the CPU below 20% and 3GB free RAM. Why would the named also failed to function properly? I don't see how httpd and named related one to another. But after I restarted the httpd, the named become normal afterwards.

Any ideas?
Thanks

Did you check your logs for specific ip addresses and/or named service logs?

You can configure an ipchain to limit incoming connections to port 80 for example, once the limit it hit (it will drop them).

You can turn on query_logging in named to view the active queries you do not want to leave this on.

You can also crank up error logging in apache to (debug) level.

Also, using this command in netfitler from the command-line: (location depending on your Linux distro...)
sudo /usr/sbin/iptstate
it will display all connections (like 'top' will be active connections)

Is this a lamp server?

If it is connection to a backend database you may need to enable caching in mysql, I had to configure several 'performance' related changes for Drupal to handle a heavy traffic usage.

Diagnosing performance related issues can be challenging, since this covers many different aspects you may need another thread on a service enhancement/modification related help.

I asked the ISP at the first whether their firewall was the one who blocking all the traffic and they say no.
I searched, tested and did many things on the server side and then I received email from them saying
"We still couldn't find any strange behaviour in our router, but we already resetting the connection. Please try again."

Why wouldn't they reset it at the first place??
Now the connection is already back to normal. Seriously normal.

My theory is still that because the traffic suddenly increases they thought it was DDOS and automatically blocking it.

Well, another good experience for me.

Thank you @rhbegin for your suggestion!

Glad I could help, dns along with web-sites can be a strange animal trying to diagnose them.

I have had dns problems in the past, and it will put grey hair on your head.