Hi,

I have 4 Linux servers and all of them are running :
- NTP service configured on /etc/ntp.conf
- DNS forwarding configured on /etc/named.conf (not actual DNS server)
- Postfix mail relay service /etc/postfix/main.cf (not mail server)
There are over 1000 VMs, which are using these servers for mentioned services. These are old Linux server and I want to decommission them and create new VM with same hostname and IPs and configuration.
I am not concerned about NTP and Mail relay, because those service will not be impacted while I will be doing cut-over from old server to new server. I am concerned about DNS service. By default, if any query comes from client, it will check from first DNS server mentioned in /etc/resolv.conf and then it will fail, without going to next server in sequence.
One option I can think is, I can use DNS service. Here is high level I can think :
- Create CNAME entry pointing from server1 to server2 and remove server1 DNS records. Now if any client tries to send query to server1, it will go to server2
- Power-off server1
- Power-on new server1 and retain same IP and hostname as original/old server
- Once server is online and remove CNAME DNS entry and re-create DNS records as original one. As soon as maps will push, CNAME will go away and queries will redirect to new server-1 again.
- Once all three services are validated, repeat same steps with server-2, server-3 and server-4.

Now one complication is, if /etc/resolv.conf is having FQDN, above will work with CNAME creation. Some of the clients may have just IP address of DNS in their /etc/resolv.conf. How can I deal with that ?

Please advice, if I am on correct theoretically, or I am missing something.

Thanks

By default the resolver should try the second entry in resolv.conf have you verified your unusual behavior with test VMs?

If it was me, my approach would be to build the new DNS server as "#3" on a different IP address. Then TEST THE HELL OUT OF IT!

I'm guessing you never implemented a configuration management solution, like Puppet or Salt. If you did, then use it to change the resolvers of your test group of VMS to use #3 and check the results.

If you haven't a config management tool (and with "1000 VMs" why not?) then pick a handful of your 1000 VMs where you've decent access to them and point them at #3 and check the results.

When you're happy that #3 is working the way you hoped then shut down #1 and give #3 the #1 IP address as a primary IP with the #3 IP address as an alias.

Then remove the old #3 IP address from your testing VMs and replace it with the original #1 IP.

Yes, moving to new IP adresses is more work but without any disturbance.

If you want to keep IP addresses then I would just shutdown/start, one server at a time. Not bothering with temporary CNAME. And announce a disturbance.
But I would rely on the second entry in the resolv.conf, then the disturbance is just a time-out. It must continue with the second entry!

If I move DNS forwarding service to new VM, with new IP/hostname, I will have to update /etc/resolv.conf of ALL clients. Yes, using ansible is one way to achieve.

That is how, I migrated non-prod environment. And now this is production. Management is asking, if there is any way, we don't have to see any resolution failure with no downtime? I am assuming, during cut-over of those 10 minutes, when I will swap hostname/IP, some applications or URLs may send query and it will query first server in /etc/resolv.conf, and resolution will fail. Is my expectation correct ? If that is only best way, then I can say, "there will be some resolution failure for few minutes".

Alternatively one way I can think is, I can unplumb IP from DNS server-1 and plumb it on DNS server-2. Then all queries will hit second server and I can work on first server. Can work this way ? There will be little downtime when I do ifdown and ifup, but less than other method.

The off/on method on the same IP is most simple.
LAN switches and switching routers cache the layer 2 addresses, that should be less than a minute.
Nothing to be changed on the clients, no risk of a deadlock (like not working ansible due to dns failure).

1 members found this post helpful.