Please help me get out of a BIND! I am suffering a bad case of DNS outage -

stardotstar · 07-06-2009, 07:25 PM

I have tried to do everything right but I know how famously tricky BIND is and how a simple '.' or absence thereof can completely screw an otherwise coherent setup.

Here is the scenario (complete with the critical complication) Thanks to any DNS expert who can give me some guidance - it is desperately needed!

1 I had a perfectly working Gentoo Linux Server running Bind as a primary Nameserver on eth0 and secondary bound to eth1.
2 I was serving a range of virtual hosts from the apache server on the box for a very long time this way.
3 when I setup a new site I made a zone file and apache vhost to use the local nameserver and then insured that the nameserver was properly populated at the registrar.

so far so good

4 My co-lo host needed me to change IPs from a few nodes within a large class c network to a dedicated subnet.
eth0 was 119.63.202.186
eth1 was 119.63.202.187
subnet mask was 255.255.255.0
gateway was 119.63.202.1

now I am configured to:
eth0 202.130.34.115
eth1 202.130.34.116
subnet mask 255.255.255.248
gateway 202.130.34.113

Happily I can ssh into both these IPs
and iLO is working on the new subnet.

so I have server access and a fallback

Please note that I was encouraged to do this by binding the new IPs to the ethernet adapters and running both the new and legacy addresses in tandem until propagation had gone through; then remove the legacy address before they became inactive.

I tried this and although I could configure my adapters to respond to both IPs I could not get my default routes to work adequetely so I elected to live with an outage while the DNS propagation happened - and now the fear begins:

5 Having access the the server at all times and now running on the new subnet I began working on the zone files and named config to bind it all to the new addresses.

6 I believe I have done all this correctly and for a time last night after a long period of outage the sites began to go live! This morning however none of them work. Something has gone very wrong.

7 The only thing I may not have in place is the reverse DNS records on the gateway - I am calling the Co-Lo into action as I write this, in case that is all it is...

8 I really need an expert to quickly run me through a few tests to see what is working and where the problem lies.

9 What I know is this:
The zones all resolve from the localhost
The apache server is pointing to the new IP as the virtual host
named and apache2 have been restarted.

Any and all help is appreciated. I insisted on doing this myself because I am hell bent on learning my way around DNS (apache and mail and MySQL and on and on and on - for some reason :shock: )

Will

xeleema · 07-06-2009, 08:33 PM

Greetingz!
Thanks for the detailed listing, however, it's a bit difficult to determine your exact problem. From reading and re-reading your post, it seems like your DNS server(s) are online and answering DNS queries when you run them on the DNS server itself, however none of your virtual sites are showing up; possibly due to a routing issue on your DNS server, preventing it from answering queries from outside it's network.

Assuming I have that right, here's what I would suggest;

1) Run a query against your DNS server (202.130.34.115 and 202.130.34.116) from an outside source.
When I try it, this is what I get;

First DNS Server

$ nslookup
> server 202.130.34.115
Default server: 202.130.34.115
Address: 202.130.34.115#53
> asdf.com
Server: 202.130.34.115
Address: 202.130.34.115#53
** server can't find asdf.com: REFUSED
> exit

Second DNS Server

$ nslookup
> server 202.130.34.116
Default server: 202.130.34.116
Address: 202.130.34.116#53
> google.com
Server: 202.130.34.116
Address: 202.130.34.116#53
** server can't find google.com: REFUSED
> exit

2) Ping-test them (assuming whatever firewall / IPfilters you have in place allows ICMP packets).
Here's what I get when I try your DNS servers (New IPs);

First DNS Server

$ ping 202.130.34.115
PING 202.130.34.115 (202.130.34.115) 56(84) bytes of data.
^c
--- 202.130.34.115 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4005ms

Second DNS Server

$ ping 202.130.34.116
PING 202.130.34.116 (202.130.34.116) 56(84) bytes of data.
^c
--- 202.130.34.116 ping statistics ---
90 packets transmitted, 0 received, 100% packet loss, time 88980ms

First things first; check your routes!

Two commands that should print out your routing table;

route -nNvee

netstat -r

(NOTE: the 'netstat' command may not support this option in your Linux distribution, the version that Gentoo deploys escapes me a the moment. I have 'net-tools 1.60' which comes with 'netstat 1.42')

If you need to add a default route, using the route command;

route add default gw 1.2.3.4 eth#

To drop an old gateway route you see listed in the output of one of the aforementioned commands;

route del default gw 1.2.3.4 eth#

WARNING: I strongly suggest that you read the man pages that come with your Linux distribution. If you drop the wrong route before a correct route is setup, you may lose access to your server. However, if you have "console" access (via a serial port perhaps) to your server, then you should be fine.

Once you've hammered out the issue, you'll need to make sure your new configuration can survive a reboot. This means updating the correct configuration files.

If I recall correctly, Gentoo has a quaint little file (/etc/conf.d/net) that you'll have to update with all the correct information.

Hope this helps!

P.S: If all else fails, try the Gentoo docs for networking.

stardotstar · 07-06-2009, 08:46 PM

Thank you that is great support - and much needed encouragement.

I have had a dig around (excuse the pun) and in conversing with a colleague it seems that the glue is missing for the primary nameservers!

My registrar does not provide me with a control to update the primary and secondary authoritative name servers and therefore I cannot tell the tld what IP is authoritative for my domains - therefore none of the virtual domains know that they must update the world...

Maybe you can confirm this line of thinking with me:

the nameservers should be

ns1.sourcepoint.com.au --> 202.130.34.115 (legacy/old 119.63.202.186)
ns2.sourcepoint.com.au --> 202.130.34.116 (legacy/old 119.63.202.187)

and they should be authoritative for (but not limited to) www.archeli.com.au
www.sourcepoint.com.au
www.arcplane.com.au
www.arcplane.com.au

can you do some dns checking and see if this theory makes sense?

Will

xeleema · 07-06-2009, 09:35 PM

Hm, good call. It may be an issue with your Registrar's NameServer entries for your Domain Name. (Can you provide their name?)

As for you're virtual hosts, you have restarted Apache since the DNS change, correct?

Some quick good news, though; I am able to get to the following sites -
www.archeli.com.au
www.sourcepoint.com.au
www.arcplane.com.au

Maybe it took a while to propogate?

Any other virtual hosts not coming up for you?

cmdln · 07-06-2009, 09:58 PM

Quote:

Originally Posted by stardotstar

Maybe you can confirm this line of thinking with me:

the nameservers should be

ns1.sourcepoint.com.au --> 202.130.34.115 (legacy/old 119.63.202.186)
ns2.sourcepoint.com.au --> 202.130.34.116 (legacy/old 119.63.202.187)

and they should be authoritative for (but not limited to) www.archeli.com.au
www.sourcepoint.com.au
www.arcplane.com.au
www.arcplane.com.au

can you do some dns checking and see if this theory makes sense?

Code:

dig ns1.sourcepoint.com.au

; <<>> DiG 9.5.1-P2 <<>> ns1.sourcepoint.com.au
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63638
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;ns1.sourcepoint.com.au.		IN	A

;; ANSWER SECTION:
ns1.sourcepoint.com.au.	604760	IN	A	202.130.34.115

dig ns2.sourcepoint.com.au

; <<>> DiG 9.5.1-P2 <<>> ns2.sourcepoint.com.au
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14734
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;ns2.sourcepoint.com.au.		IN	A

;; ANSWER SECTION:
ns2.sourcepoint.com.au.	604800	IN	A	202.130.34.116

dig -t NS www.sourcepoint.com.au

; <<>> DiG 9.5.1-P2 <<>> -t NS www.sourcepoint.com.au
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44996
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;www.sourcepoint.com.au.		IN	NS

;; AUTHORITY SECTION:
sourcepoint.com.au.	10800	IN	SOA	dns1.sau.net.au. support.serversaustralia.net.au. 2009070702 28800 7200 1209600 604800

;; Query time: 234 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Mon Jul  6 21:53:06 2009
;; MSG SIZE  rcvd: 114

dig dns1.sau.net.au

; <<>> DiG 9.5.1-P2 <<>> dns1.sau.net.au
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 52371
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;dns1.sau.net.au.		IN	A

;; ANSWER SECTION:
dns1.sau.net.au.	14256	IN	A	202.130.32.10

dig -t NS sourcepoint.com.au

; <<>> DiG 9.5.1-P2 <<>> -t NS sourcepoint.com.au
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12824
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;sourcepoint.com.au.		IN	NS

;; ANSWER SECTION:
sourcepoint.com.au.	604800	IN	NS	dns1.sau.net.au.
sourcepoint.com.au.	604800	IN	NS	dns2.sau.net.au.

To me it looks like sourcepoint.com.au authoritive nameserver is dns1.sau.net.au (202.130.32.10)

But ww.arcplane.com.au is pointing at ns1.sourcepoint.com.au.arcplane.com.au which has no A record

Code:

dig ns1.sourcepoint.com.au.arcplane.com.au

; <<>> DiG 9.5.1-P2 <<>> ns1.sourcepoint.com.au.arcplane.com.au
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 49437
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;ns1.sourcepoint.com.au.arcplane.com.au.	IN A

;; Query time: 1 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Mon Jul  6 21:57:33 2009
;; MSG SIZE  rcvd: 56

stardotstar · 07-07-2009, 07:45 PM

Thanks Nick. Things have begun to resolve themselves since it seems to have all been about propagation and stale caches. I still can't see half my sites from my work network but at home on optusnet cable it is fine - people I know with optusnet adsl can't see it but the cable network is fine...

BTW there must be some confusion becasuse there is no ns1.sourcepoint.com.au.arcplane.com.au.
There is arcplane.com.au which is name served by ns1.sourcepoint.com.au and archeli.com.au etc
the problem seems to have ended up being a lack of glue for the new IPs and ns1.sourcepoint.com.au - being a registrar customer of a planet domain reseller I couldn't update the glue records and therefore no way of telling the world that my ns1 and ns2 had new IPs... We tried to fix this (more or less successfully) by making the serversaustralia name servers authoritative for ns1 and ns2 sourcepoint.com.au

Not sure why wider propagation has not gone down but I am hoping that it will in good time.

Will

cmdln · 07-07-2009, 11:08 PM

Quote:

Originally Posted by stardotstar

Thanks Nick. Things have begun to resolve themselves since it seems to have all been about propagation and stale caches. I still can't see half my sites from my work network but at home on optusnet cable it is fine - people I know with optusnet adsl can't see it but the cable network is fine...

Yeah the state of DNS is pretty bad all around. You can use a reverse proxy on the old ips to pass the traffic through to the new ips if you want. Since your not moving machines just having apache listen on both the old and new ips would be the easiest thing.

I had a similar issue when I moved a fairly busy website.
http://www.cmdln.org/2008/11/01/trus...ove-a-website/