nslookup: connection timed out; no servers could be reached

dkm999 · 06-26-2008, 06:17 PM

Sorry about the syntax error; I missed an '&&' between the 'udp port 53' phrase and the 'host 64.81.79.2' phrase.

The goo news is that this gets us a little further. If I take the liberty of rearranging the lines a little, to match up the requests and replies, we get this:

Code:

15:00:10.159128 IP 64.81.79.2.53 > 207.7.135.73.40301:  51927 2/0/0 A 206.190.60.37, (59)
15:00:10.159321 IP 64.81.79.2.53 > 207.7.135.73.40302:  28291 NXDomain* 0/0/0 (43)
15:00:10.159894 IP 64.81.79.2.53 > 207.7.135.73.40304:  51541 NXDomain* 0/0/0 (43)

15:00:10.161585 IP 207.7.135.74.57781 > 64.81.79.2.53:  63784+ AAAA? glocap.com. (28)
15:00:10.164619 IP 64.81.79.2.53 > 207.7.135.74.57781:  63784 0/0/0 (28)

15:00:10.164778 IP 207.7.135.74.57781 > 64.81.79.2.53:  65417+ A? glocap.com. (28)
15:00:10.167807 IP 64.81.79.2.53 > 207.7.135.74.57781:  65417 1/0/0 A 207.7.135.70 (44)

15:00:10.169751 IP 207.7.135.74.57781 > 64.81.79.2.53:  50096+ AAAA? yahoo.com. (27)
15:00:10.172958 IP 64.81.79.2.53 > 207.7.135.74.57781:  50096 0/0/0 (27)

15:00:10.172115 IP 207.7.135.73.40305 > 64.81.79.2.53:  15233+ A? a.mx.mail.yahoo.com. (37)
15:00:10.174877 IP 64.81.79.2.53 > 207.7.135.73.40305:  15233 1/0/0 A 209.191.118.103 (53)

15:00:10.172753 IP 207.7.135.73.40306 > 64.81.79.2.53:  13242+ AAAA? glocap.com. (28)
15:00:10.175876 IP 64.81.79.2.53 > 207.7.135.73.40306:  13242 0/0/0 (28)

15:00:10.173145 IP 207.7.135.74.57781 > 64.81.79.2.53:  59041+ A? yahoo.com. (27)
15:00:10.176094 IP 64.81.79.2.53 > 207.7.135.74.57781:  59041 2/0/0 A 206.190.60.37, (59)

This shows how the protocol is supposed to work. Note near the bottom of this list, how 207.7.135.73 asks for both an A record and an AAAA record, and gets back a positive reply for the A record (1/0/0) and a negative one for the AAAA record (0/0/0).

Unfortunately, none of these conversations has anything to do with xmail1, whose address, according to previous posts is 207.7.135.71. Let's try again on the tcpdump command:

Code:

#tcpdump -i eth1 -nn udp port 53 && host 207.7.135.71

This should show us the conversation between xmail1 and the DNS server(s).

kbighorse · 06-27-2008, 12:31 PM

I did another experiment. I unplugged 192.168.1.19 from the local subnet and rebuilt xmail1 to be 192.168.1.19, with the same hostname and everything. I had the same problem. I switched back and 19 worked fine as before, so I would think that eliminates the firewall possibility.

I also upgraded to Fedora Core 9 to eliminate the OS as a source of the problem, and installed Ubuntu Hardy Heron as well and had the same problem both times.

I'm thinking that leaves hardware issues, but in every case, local subnet connectivity works fine.

dkm999 · 06-27-2008, 01:37 PM

I am not sure that your experiment proved anything except that there is some configuration problem on xmail1 (which we thought likely anyway). The point of doing the tcpdump trace on trusty was to confirm that the DNS reply packet was indeed getting returned.

Once we have established that the return is happening (and that the tracing process actually shows the behavior we expect), we can apply that same tracing process to xmail1 to make sure whether the return packet is getting to that machine or not, and after that, we can start looking at what happens to it once inside xmail1.

If this process is unpalatable to you, I apologize, but when we are trying to stalk an obscure bug, I think it is important to be clear about what we know and what we don't know before making up further experiments.

kbighorse · 06-30-2008, 12:02 PM

Already a problem:

after a lot of output for other servers:

Code:

3716 packets captured
7432 packets received by filter
0 packets dropped by kernel
Host 71.135.7.207.in-addr.arpa not found: 3(NXDOMAIN)

I'm checking it out, but here's from my laptop:

Code:

Cornplanter:~ kimball$ dig xmail1.glocap.com

; <<>> DiG 9.4.1-P1 <<>> xmail1.glocap.com
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 883
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;xmail1.glocap.com.		IN	A

;; ANSWER SECTION:
xmail1.glocap.com.	7200	IN	A	207.7.135.71

;; Query time: 19 msec
;; SERVER: 68.87.76.178#53(68.87.76.178)
;; WHEN: Mon Jun 30 09:52:18 2008
;; MSG SIZE  rcvd: 51

farslayer · 06-30-2008, 08:27 PM

Definitely some issue...

Code:

debianetch:~$ telnet xmail1.glocap.com 25
Trying 207.7.135.71...
connection timed out

debianetch:~$ telnet xmail2.glocap.com 25
Trying 207.7.135.72...
Connected to xmail2.glocap.com.
Escape character is '^]'.
220 xmail2.glocap.com ESMTP Sendmail 8.13.4/8.13.4; Mon, 30 Jun 2008 21:18:23 -0400
quit
221 2.0.0 xmail2.glocap.com closing connection
Connection closed by foreign host.

I can not connect to xmail1 manually either..... (the server is up and running right ?)

kbighorse · 07-02-2008, 11:20 AM

Yes, xmail1 is up and running. Totally not sure what's going on.

farslayer · 07-03-2008, 03:57 PM

So if you can telnet to port 25 of xmail1 from inside your network,. then there is something blocking or not forwarding the ports between that server and the Internet.. or the other direction..

Personally I would setup Wireshark on a laptop and sniff inside and outside your firewall for the traffic..

kbighorse · 07-08-2008, 10:56 AM

Yeah, here's some output from within the network:

Code:

[root@trusty ~]# telnet 192.168.1.21 25
Trying 192.168.1.21...
telnet: connect to address 192.168.1.21: No route to host
telnet: Unable to connect to remote host: No route to host
[root@trusty ~]# ping 192.168.1.21
PING 192.168.1.21 (192.168.1.21) 56(84) bytes of data.
64 bytes from 192.168.1.21: icmp_seq=1 ttl=64 time=0.148 ms
64 bytes from 192.168.1.21: icmp_seq=2 ttl=64 time=0.097 ms
64 bytes from 192.168.1.21: icmp_seq=3 ttl=64 time=0.096 ms
64 bytes from 192.168.1.21: icmp_seq=4 ttl=64 time=0.097 ms

--- 192.168.1.21 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2998ms
rtt min/avg/max/mdev = 0.096/0.109/0.148/0.024 ms
[root@trusty ~]# dig 192.168.1.21

; <<>> DiG 9.3.2 <<>> 192.168.1.21
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 20713
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;192.168.1.21.			IN	A

;; AUTHORITY SECTION:
.			10800	IN	SOA	A.ROOT-SERVERS.NET. NSTLD.VERISIGN-GRS.COM. 2008070800 1800 900 604800 86400

;; Query time: 74 msec
;; SERVER: 209.213.223.118#53(209.213.223.118)
;; WHEN: Tue Jul  8 10:06:50 2008
;; MSG SIZE  rcvd: 105

I need to understand better what telnet, nslookup, dig, ping etc. each offer and indicate, i.e. why some work and others don't. I'm checking out Wireshark in the meantime.

farslayer · 07-09-2008, 11:35 AM

This guide looks like it touches on those tools and more while providing some examples..

http://www.linuxhomenetworking.com/w...roubleshooting

it doesn't cover dig, but dig does pretty much the same thing nslookup does (just better imho)

mstephensonUL · 07-09-2008, 01:41 PM

This might be a longshot but you maybe you could try a dump and restore from a working system to this one. If it works you still won't know what the configuration problem was but it might help you fix the immediate problem. You might be able to do this with tar as well.

http://www.linuxscrew.com/2007/08/13...estore-backup/

btw... you and I just spoke face to face about an hour ago

mstephensonUL · 07-09-2008, 05:57 PM

You might also check the port on the switch. Maybe it has a strange vlan assignment compared to your other ports.

mstephensonUL · 07-11-2008, 02:46 PM

I saw an issue similar to this yesterday involving a netscreen firewall. Source nat translation wasn't working for this server in the 1918 address space. It didn't have a publicly mapped IP address and for some reason the source nat was busted. I verified this by running tcpdump on an internet server and then attempted telneting to it from the problem server. I saw the private 1918 address hitting the internet server so the packets had no home to return to. Check your nat rules in iptables on both your firewall and your problem server as well as a working server. Maybe something will pop out at you.

iptables -L -t nat

I fixed this issue by assigning a dynamic ip pool in netscreen and built a policy to perform source nat translation and pull an IP from this dynamic pool. This way the internet server had a public address to route back to. Don't know how you would do this in iptables