I've been working all morning on a crazy problem and I'm out of ideas. I have a Slackware64 14.2 domain controller which does DNS, DHCP and mail serving. It has 2 NICs. eth0 is Internet facing. No problem there. eth1 is for the LAN. Sometime before this morning the domain controller lost the ability to communicate with all hosts on the LAN, except for one! Trying to ping or connect to any host from the domain controller gives "Destination Host Unreachable". Nor can any host connect with any other host on the LAN.

I'm suspecting a bad eth1 card, but it can connect with one of the hosts, and only one. Which makes me doubtful that it's the NIC.

I've taken down the firewall except for the forwarding to eth1. I've powered down and rebooted hosts and domain controller. I've cycled power on all switch. Nothing help.

Any ideas?

What does arp -an show when this happens? And does tcpdump -i eth1 arp show ARP requests being sent and received when you ping hosts on the LAN?

If you see ARPs being sent but no replies, it could be a broken switch failing to flood broadcast frames.

Solved the mystery. After swapping network cards and replacing two of the switches (still nothing worked), I disabled all connections except between the domain controller and one host, and started going through all the related settings. I found that /etc/dhcpd.conf had a corrupted line and dhcpd stopped processing the config at that point. That explains why one of the hosts was able to connect, but not others: the connectable host's settings occurred before the corrupted line. When I fixed the dhcpd.conf, everything was back in action! If I had thought to look at dhcpd messages and config first, I'd have saved myself about 4 hours of pain and the office 2/3 of a day of lost productivity. But, that wasn't among my initial suspicions.

How did dhcpd.conf get corrupted? My guess is that I was examining this file remotely in an editor without specifying read-only mode and, when the connection dropped, bogus characters got written to the file. This has happened infrequently before with other programs and scripts, but I'm usually right there when it happens and, after a bit of cursing, can reconnect and edit the file to remove the noise. To further obfuscate things, this file corruption probably happened several days in the past and did not affect things until the DNS and DHCP servers restarted per normal logrotate operations on Sunday.