Hello,

I am working on a DR plan for our data center which includes our DNS servers and LDAP servers.

Currently, we have a master/slave DNS server and LDAP server on a single subnet. What we need to be able to do is...if the data center were to be completely destroyed (that's the scenario I was told to plan for) we would need to maintain DNS and LDAP services in another subnet at a different physical location.

I have looked around and have found several options:

Option 1:

Have a master DNS server at the DR site and use schedule rsync or scp to synchronize the zone file.

Option 2:

Set up the DR DNS server as slave and promote it to master in the event of a disaster.

Both of those options are reasonable, but I'm looking for a better option. I'd like to find a way to have the main DNS master and the DR DNS master to stay in sync without having to create custom, scheduled scripts to copy over the zone files.

I guess, basically, I want to have a second DNS master that acts as a slave in the sense of staying in sync with the master, but then automatically kicks into "master" mode if the main server is lost so that I can add/remove/change DNS entries on the DR server to get our systems back online. Sort of a active/active DNS server.

As for LDAP, we have essentially the same scenario. We have a master/slave on the main subnet and want a syncrhonized master at the DR site.

Does anyone know of a way to do this? Is it even possible? Or is there an even better way to handle this scenario?

Hi there,

Herewith some thoughts on the DNS aspect of your question.

A few quick questions first:
* Do you have only these 2 locations & subnets, or is there a larger LAN / WAN involved?
* Is the DR site used only for DR, or is there a need to access that network at other times too?

In my opinion you have one of 2 options at the DR site in case of switchover, namely:
* Change device IP addresses (and routing config for a WAN) to match the original production IPs, in which case no DNS entry changes are required, or
* Change DNS entries to let known names point to new IP addresses, in which case no IP or routing changes should be necessary (assuming adequate redundancy to still allow all sites to reach you).

The first option above is probably more complex to implement, and thus not recommended in most cases. If you go this route, your slave DNS needs to take on the IP of the original master, and be promoted. No individual DNS entry changes should be required, though.

The second option is easier to implement, as it requires only DNS changes, meaning one place to make changes rather than many devices to reconfigure. The DNS entries will have to change, though, which means switching over isn't just as simple as just promoting the slave to master.

I can suggest two possible approaches to make the required DNS changes a little easier, namely:
* If you keep the host part of your IP addresses the same at both sites (on their respective subnets, i.e. for example DNS server on 192.168.100.2 in PROD and 192.168.200.2 in DR), you should be able to write a relatively simple script to substitute one subnet for another in the zone files (i.e. use a command like "sed" to substitute 192.168.100 for 192.168.200 in this example).
* If you maintain DNS entries for your DR servers in a different domain, you can switch over to DR with the same zone files, and just minimal changes to named.conf and possibly SOA records to change the domain name.

Regarding automatically switching over if the master server is lost, I think this would be tricky, as:
* Some problems (for example a network failure between the 2 sites, or a power failure) are easy to fix in the prod site and/or do not warrant switching to the DR site.
* Detecting a failure, and distinguishing between a real disaster and a lesser problem as mentioned above, is a challenge.
* Switch back may be harder to automate, and may require at least as much work as using manual scripts to do the initial failover.

Just my 2c - hope it provides some food for thought :-)

Clifford,

Thanks for the response.

In answer to your first two questions...

1. These are the only two subnets involved. The PROD environment is our DMZ and houses much of our web site servers and supporting services (e.g. DNS & LDAP). The DR environment is a brand new DMZ that we are building out. This will be for....

2. Our Staging environment which, in the event of a disaster, we will promote to our PROD environment. So, most of the time (hopefully, ALL of the time) this environment will be used for staging for our web site(s). If we lose the PROD DMZ, we want to fail over to the staging DMZ.

As far as host names, what I was thinking was that we'd have all systems named the same except that we'd prefix the DR/Staging servers with some designation to distinguish them from them from PROD. (e.g. APPSVR1 --> STGAPPSVR1)

Unfortunately, the DR DNS server has to be master because there would be a couple of changes we would need to make to the DNS tables if we had to fail over, so it can't be a slave.

To me, so far, it seems like the cheapest/easiest/lowest maintenance way to handle this is to just have two masters and use a scheduled rsync or scp to keep the zone files up to date. I was told by one of our senior staff that this was a reasonable solution but that I should try to find something better. It sounds like, if nothing better is to be found, we'll probably proceed with that.

We definitely are going to have a similar IP scheme such as you suggest (adding 100 to the third octet) to try to make things less complicated. Since we're using our staging environment, that should minimize DNS table changes, though there will still be a few since CC auth requests will have to go through a production system, we'll need to point to a prod DB, etc. But hopefully, the number of changes will be minimized.

Also, since it's a staging env, what we'd probably do is have all entries for both DMZ's available in the PROD and DR DNS tables.

What are your thoughts on my proposed rsync method? I know we'd also have to figure out how to ensure that we do a 'service named reload' if the zone file gets updated, but I have a couple options with that, as well...

1. I could set up the sync script to pull the zone file from PROD and include a line to reload the zone file after each update.

2. I could set up the sync script to push from PROD and do another cron job to reload at a designated frequency.

3. Since I'd have to make some manual changes to the zone files in a DR situation, I'd just do the reload manually at that time.