Troubleshooting Denial of Service
 

I have a Centos 4.4 server that is a standard LAMP server and as of recently I have had the server deny all services in the mornings but not every morning consistently. When I look at the physical server, the nic card light does blink but no so much that it is out of the ordinary, the hard disk activity light blinks too but is not solid or excessive, the server seems to be up but it denies every incoming connection other than ping requests. No dns, web, or even SSH

I thought it must be a cron job hogging system resources so I analyzed the cron log and the cron tasks seem to be running properyl even when the server is non responsive to inbound services. I ran every scheduled cron job and non of them bring the server down.

I have looked at the server logs but there is too much info to figure out what could be the problem and what couldn't. Is there a specific type of logging that I need to set or other method that I can use to see what is bringing my server down at night?


BTW, a simple hard reboot solves the problem until the next night/morning.

Your help is appreciated.

Don't rely on lights on a NIC card. I'm assuming this server is accessible to the outside world, what happens when you can't reach it if you try to pull up one of the services locally if possible?

I am on the same local subnet as the server and I cant access it from the LAN either so its not a router/firewall issue. I just setup service monitoring on the ssh service (on of the inaccessible services) so that I will be emailed and the server will reboot if SSH goes down. It won't tell me why the services are failing but at least it will keep my server up and hopefully I will have something in my syslog just before the reboot that tells me something about why SSH crashed.

Any other advise is welcome.

http://linux-mm.org/OOM_Killer ?

Thanks Chris. I gave it a look and it's OK. I think it is a good solution for servers that can only support a small amount of memory compared to the resources required by the services running on the server. If I knew what service(s) were hogging my resources and causing the Dos then I would try to nice them to the lowest priority and see if that fixes it before I move to more drastic measures. Perhaps I can use your idea to log the process that it chooses to terminate if it has logging abilities.

Thanks again.