Life eating network problem
 

Hi there

Sure hope someone can solve this one, will save me an 8 hour round trip, but more importantly hours of my life.

PROBLEM

[root@mediasrv01 ~]# ping www.bbc.co.uk
ping: unknown host www.bbc.co.uk

Essentially this, but not quite.....

[root@mediasrv01 ~]# nslookup www.bbc.co.uk
;; connection timed out; no servers could be reached

[root@mediasrv01 ~]# nslookup www.centos.org
Server: 213.248.100.54
Address: 213.248.100.54#53

Non-authoritative answer:
Name: www.centos.org
Address: 72.232.194.162

[root@mediasrv01 ~]# host www.mysql.com
;; connection timed out; no servers could be reached
[root@mediasrv01 ~]# host www.ebay.co.uk
;; connection timed out; no servers could be reached
[root@mediasrv01 ~]# host www.booktribes.com
www.booktribes.com has address 195.12.232.9
[root@mediasrv01 ~]# host www.dell.com
www.dell.com is an alias for www1.ins.dell.com.
www1.ins.dell.com has address 143.166.224.244
[root@mediasrv01 ~]# nslookup www.myspaced.com
Server: 213.248.100.54
Address: 213.248.100.54#53

Non-authoritative answer:
www.myspaced.com canonical name = myspaced.com.
Name: myspaced.com
Address: 68.178.254.123

[root@mediasrv01 ~]# nslookup www.myspace.com
;; connection timed out; no servers could be reached

OK maybe the name server .... well nope, changed to another on a different ISP network.

And do a:

service network start

first then....

Notice the spelling mistake in MYSQACE

[root@mediasrv01 ~]# nslookup www.mysqace.com
Server: 195.12.1.1
Address: 195.12.1.1#53

Non-authoritative answer:
Name: www.mysqace.com
Address: 66.45.252.237
Name: www.mysqace.com
Address: 66.45.252.236

[root@mediasrv01 ~]# nslookup www.myspace.com
;; connection timed out; no servers could be reached

But then....

[root@mediasrv01 ~]# nslookup www.mytube.com
;; connection timed out; no servers could be reached

AND FROM A DIFFERENT machine...

nslookup www.mytube.com
Server: ptn-cdns02.plus.net
Address: 212.159.6.10

Non-authoritative answer:
Name: www.mytube.com
Address: 209.62.20.159

###############################################

So what is the setup, right the server is running CentOS 4.4 it is behind a transparent hardware firewall, I can ping out on IP and FQDN (if it resolves), I can ssh in from the public side (network does not drop and is not flaky)

Three network cards, unfortunately I have found that the setup is not that intelligent as it seems to assign the device mappings an unordered manner. Three onbaord NICS a dual 1Gb and a single 100Mb (all onboard)
eth0 is assigned to the 100Mb and eth1 and eth2 to the 1Gb NICs.

The public interface is running on eth1 and the private side is running on eth0 - due to the fact I need 1Gb connectivity on the public side, being a media server :)

I figured it may have been an issue with the eth0 and eth1 eth0 being the PRIMARYDEVICE, so I changed the modprobe.conf and changed the e100 to eth1 and the e1000 to eth0, swapped the ifcfg-eth0 and eth1 and editted the DEVICE in both and set them to the appropriate devices, this did not solve it.

So now I am stuck on intermittent but somewhat structured resolution failures, almost as if DNS name requested where hashed to work or fail...

Thanks anyone that can shed some light on this dire situation. Yes it is dire because my life is just ticking away :)

Do you run your own DNS servers for the inside, and is this the gateways nslookups or an internal hosts?

Regards,

Alunduil

We do, but they are not recursive (just PRIMARY) so we use our carriers or as in this case another public recursive DNS server, if appropriate.

Are you sure you don't use a name server that deliberately blocks some sites? What about looking up domain names using an external public DNS, like OpenDNS - http://www.opendns.com/ ? Can you resolve the hosts back (IP to domain name)?

I am sure we do not use NSs that block sites, we run on a Tier1 backbone and our carrier's DNS servers service all other machines in our data centre. This is a server box so do not have a browser on it and OpenDNS does not seem to have public recursive DNS servers

nslookup www.mysql.com auth1.opendns.com
Server: auth3.opendns.com
Address: 208.69.39.2

DNS request timed out.
timeout was 2 seconds.
*** Request to auth3.opendns.com timed-out

However, I am trying to use another public Tier2 ISPs DNS servers which we use as secondary on our network, which also has the same results.

But please do not let that stop you from any other ideas :) Because I am fresh out, actually I hav stopped trying becuase I want someone else's mind to look at the bl00dy problem :) As maybe someone will see the light :)

first check your connectivity/packet loss ping any external ip for atleast
few min. and see the reasult.

Hi soroccoheaven I was hoping some people would take an interest...

[root@mediasrv01 ~]# ping www.google.com
PING www.l.google.com (66.102.9.99) 56(84) bytes of data.
64 bytes from lm-in-f99.google.com (66.102.9.99): icmp_seq=0 ttl=244 time=21.3 ms
...
...
..
..
.
64 bytes from lm-in-f99.google.com (66.102.9.99): icmp_seq=121 ttl=244 time=20.7 ms

--- www.l.google.com ping statistics ---
122 packets transmitted, 122 received, 0% packet loss, time 125207ms
rtt min/avg/max/mdev = 20.351/22.530/75.302/7.660 ms, pipe 2


I figure that should be enough. A DNS resolution works instantly, when it works and takes the normal DNS timeout (about 10 seconds) to fail and it repeated fails every time to FQDN that do not work and instantly works on ones that do.

And the set of domains you can't resolve remains the same whichever DNS you use? What about trying to use recursive DNS queries (like resolving mysql.com while using their NS for the domain (which you can get by a query from a working box)?

Hi raskin

No joy on either FQDN or the IP of the primary NS for the domain

[root@mediasrv01 ~]# nslookup www.mysql.com dns1.mysql.com
nslookup: couldn't get address for 'dns1.mysql.com': failure
[root@mediasrv01 ~]# nslookup www.mysql.com 213.115.162.1
;; connection timed out; no servers could be reached

[root@mediasrv01 ~]#

What about resolving names using TCP DNS protocol? Maybe it will carry some status information when breaking connection. Try packet sniffers - maybe you get some valuable ICMP replies that can give information..

Hi, just a quick update. It was a faulty switch, I made the round trip and plugging into to another switch solved the problem. Thanks for all the suggestions and help.