Hi, I run a community website server running LAMP (MySQL, Perl). It's got Debian Lenny 2.6.26-2-amd64, with all the latest updates. The server has been working just fine for months, it's a popular website which gets lots of traffic every day. This morning I tried changing one line in the /etc/hosts file, as part of an attempt to improve my sendmail config. The previous initial lines read:

127.0.0.1 localhost
127.0.1.1 spidey.nilspace.com spidey

I changed it to something like this:

208.64.24.170 spidey.nilspace.com spidey
127.0.0.1 localhost

However, on reboot, the server didn't come back up on the network. The server is in a colo center, so I can't easily get to the console, but eventually they gave me a remote KVM Java app to use.

I found that the server is up, but somehow the networking just isn't working any more. I can't ping the gateway, it just comes back with 'Destination host unreachable'.

ifconfig comes back with output that looks fine. I can't copy-paste text from the KVM window, unfortunately, but it does look ok. The link appears to be up, at least as far as Linux is concerned. I can ping localhost, but not the gateway. My host at the datacenter tells me the switch thinks my server is up.

Here are some interesting clues, perhaps:

When I use 'arp', it hangs for about 20 seconds, then comes back with a single line starting with the server's local ip address, but the HWType is blank and HWAddress is "(incomplete)", and the rest of the line is blank.

When I type 'route', it prints out the first line immediately (starting with the server's own ip address), but then hangs for about 20 seconds, then prints out the second line that starts with 'default' and then has the gateway's ip address.

I have put /etc/hosts back to the way it was, and rebooted, but something has obviously changed during that reboot. I can't for the life of me figure out what happened. It actually feels like a hardware issue, but I know that this is something that always occurs to me when I can't figure something out. And why would this hardware issue arise just because I did a soft reboot? Does that happen? So it's probably a software config error, but I don't know where else to look. I've checked /etc/hosts, /etc/network/interfaces, where else could it be screwed up?

Can anybody give me some things to try out which will tell me for sure whether this is a hardware problem or a config issue? I am at a loss here.

Thanks,

Neil

I think I now know at least why 'arp' and 'route' were hanging: It was because they were trying to use DNS to resolve an ip address, and obviously there isn't any network connectivity, so that hung and then failed. If I use 'arp -n' and 'route -n' then they both return immediately, albeit with the same results.

I'm pulling my hair out here - my hosting provider says it doesn't sound like a hardware issue, but I can't think of anything else to check. And I can't ping the gateway.

I would really appreciate any clever diagnostic tests to help me figure out where this is going wrong... I'm pulling my hair out there, the server's been down all day now and I'm no closer to figuring out what's wrong. I'm getting kind of desperate, this feels like one of those horrible nightmares where nothing works and you don't know why. Anybody?

Thanks!

Neil

Here are some screenshots from my KVM session, showing some relevant info:

/etc/hosts: [image removed]

Output of ifconfig: [image removed]

/etc/network/interfaces: [image removed]

Results of ping gateway: [image removed]

Results of 'arp -n' and 'route -n': [image removed]

Anything stand out here?

Thanks!

Neil

Gee, thanks for all the help, guys.

Some closure: Apparently it was a problem with the cache in the switch at the datacenter. When they flushed the cache and reset the port, my server was able to see the network again. When I made a small change to my /etc/hosts file, it must have set up some kind of MAC address conflict (or something) which didn't go away until they flushed and reset.

Neil