Hi

A few days ago my server suddenly went offline out of the blue, after over a year of uptime. RHEL 4. I could get the ISP to ping it from the switch, but from outside the subnet ping requests were being ignored. The machine was up and running, and when I got to the console I noticed a few "Neighbour table overflow" entries on the console.

After a few hours of struggling, I eventually found that the default route in the routing table was incorrectly set to the IP of the server and not the IP of the gateway. So all packets to the subnet were getting out, but no others. My /etc/sysconfig/network file had an incorrectly set GATEWAY setting, which must have been there since I installed over a year ago (the date of the file pretty much confirms this), and I must have adjusted the routing tables by hand.

I can only think that for some reason the routing tables where suddenly reloaded, and the incorrect gateway was loaded from the /etc/sysconfig/network file. Does anyone know why this might have happened?

I think it's pretty unlikely that I've been hacked (although always possible). The firewall is configured well, very few ports open, and there seem to be no other signs of a hack - I would have expected more than a routing table change / reload in that case.

What could cause the reload? IPTables throwing a wobbly? TCP stack? Kernel? Malicious packets, ICMP or otherwise?

Any suggestions/ideas would be appreciated!

A power outage? Check uptime.

Yeah, good idea, but the machine is in a data centre and had been up for over a year...

The interesting this is that if I hadn't incorrectly configured the /etc/sysconfig/network file, I would never have known that the routing tables were reloaded - OK, to be accurate, I'm *assuming* they were reloaded, as that's not recorded in the logs AFAIK... It's the only explanation I have for why the default route was suddenly incorrect.