Hello,

I work for an ISP and we currently have 3 bonded T1's into a Cisco 2600 router. We are changing to a different provider and just going with one T1 for starters. Now, the problem is we have a full set of DNS, web, and email servers. The problem will not be so much with our customers, as our DNS will have all the new IP information once I enter it. Its the people that are trying to contact our customers or our customers websites.

So far the best solution I can think of is to move in this order:

1. Update the IP information for the NS at the registrar for our domain name
2. Change the DNS servers over to the new T1 and set the new IP information etc.
3. Create a dummy server that will have eth0:x interfaces for all the old IP addresses, using iptables to dnat the packets that are going to those old IP's to the new IP's. (I am under the assumption I should not have to SNAT since all the IP's we are talking about are public IP addresse).
4. Move all the servers to the new T1 and set the new IP information in the servers.
5. In 3-5 days, turn down the dummy server since 99% of the world will have the updated DNS information.

So I need to know if I am on the right track here as far as downtime and DNS issues go... will this method eliminate most (if not all) downtime? Other than the time to physically move the server connections from one router to the other and set the new IP's of coarse. And am I correct in my understanding of using dnat through iptables to forward traffic?

This is what I have found that looks like it should work:
iptables -t nat -A PREROUTING -i eth0 -d old.ip.address -j DNAT --to-destination new.ip.address

Thanks in advance for any advice/tips.

i would suggest just using a short TTL on the DNS query. if for a few days before the migration you whip the TTL of the domains to 10 minutes or so, then all valid DNS servers caching those A records on the net will timeout within 10 minutes. so when you do a snap change, your connections should be sorted and customer connectivity back within an hour max. obviously this depends on the other name servers honourin TTL's, but most will to not worry about it. if they don't honour it somewhere, then you're still only looking at a day for that last 1% of users.

Thanks for the input. I am assuming you are just talking about the TTL in the zone file yes? Would we have to do this for every domain we host as well? I don't think I made it clear that we host quite a few customer domains as well, and if that is the case, that route would be nigh impossible with the amount of manpower we have.

If someone could verify whether the dummy box idea would work, and/or if the TTL would have to be changed on all domains we host as well as our own, I would appreciate it.

Thanks again.

if you're using bind then check this... http://www.netadmintools.com/art232.html

i don't think you should look to go to the lengths you're going to, it's not a resonable thing to be expected to do. you could well have issues with asymmetric routing though, as you have a firewall before this server that you want to dummy the ip addresses on? well that will only see half a conversation so should be rights complain like there's no tomorrow. only poor firewall rules should really resolve that, which isn't a good direction to head in.

You're migrating to new internet links? just the link from your servers to the net? well what's stopping you just using both links at the same time? the old firewall isn't going to see any difference other than less traffic until it all dissapears anyway. to combat asymetric routing in that case, i'd just simply source nat traffic as it enters the network on the old links to ensure it returns to the net on that link two, despite the default route having been pointed towards the new one.

may i ask how and why an ISP is in this situation? What sort of ISP only has that much networking?

A small, local ISP

I am mainly exploring my options and trying to find the best route in terms of maintaining uptime as much as possible without having to shoot myself to do it

I did not design the setup, but I don't see a way to use both T1's without spending a bunch of money on a multi-wan ported router (not an option). The one multi-wan router we do have is filled with bonded 3-T1 setup (our current connection). I wouldn't even ask if that was possible as that would be an easy solution.

As far as asymmetric NAT, I think I understand your point about the firewalls. However, I am talking of using NAT at the server level to forward traffic via a dummy box (server), that will have all the current IP's assigned to it, to the new IP's that the servers will have once i move them to the new T1. All servers have public IP's, and the firewalls are dealt with at the server level (no third party firewall at the moment between the servers and the T1 routers), so I don't think the firewall is really an issue for us, unless I am not understanding something (which is completely possible as I am not an expert in linux... yet ).

Let me know if this makes sense or if I am on the right track, or if what I am thinking is feasible, or what...

Thanks again.

I guess there are three possible solutions:

1. Set TTL to really low and wait a few days then set it back after the move. This isn't workable if I have to change the zone files for ALL domains though... if someone can confirm whether thats the case or not...

2. Use iptables to rewrite the destination of packets to the old ip's to the new ones at the server level, on a dummy machine. I need some clarification to tell if this is actually feasible tho:

- Dummy machine is loaded up with all the servers' current IP's
-- Traffic A arrives at Dummy server. Destination is NAT'ed to the new server A.
-- Traffic A arrives at New server A
--- Question 1: does Traffic A still contain the original "source ip" after the initial DNAT at Dummy server?
--- Question 2: In order to reply to the original source and have the original source recognize the data as its reply, does new server A have to SNAT the packet "source" as if its coming from the dummy server so the original source will recognize it?
--- Question 3: Assuming 1 and 2 are true, how could new server A recognize and SNAT only packets that were DNAT'ed from the dummy server, but leave all natural traffic alone? Or is this possible?

If any 3 of my questions are false, then this would appear unworkable to me...

3. The best option if I can convince the boss to do it: buy a dual wan port router, and/or determine if I can disconnect one of the 4 cables in the 4 T1 slots on the back of our Cisco and program it to the new T1.


Obviously 3 is the "best" option... but we are a very small business and it may be difficult to convince the boss to do it (unless there isn't any other feasible option).

Thanks again for any advice and help.

how does your 2600 fit in? are *all* wan links connected to it? if that were the case, and there are no stateful devices on the outside of that router, then i'd be wondering if you actually have any notable problem at all. whichever IP is connected to will reach that router, and on that router you then have a place to add 1:1 nat's for each new ip, or if you're just holding the public IP on each box, as opposed to a more conventional approach on your scale of a nat layer within the network, then why not just put both ip's on each server. as above, after switch over, it still wouldn't necessarily matter either way how traffic reaches you.

don't forget that you can do a *LOT* of *very* funky networking with open source too... an install of zebra or quagga will give you an IOS-u-like interface and 95% of general IOS functionality within any artbitrary hardware and network interfaces.

They were set up to hold the public ips on the boxes themselves. The router is something of a passthru as it were, but sits as the Gateway. The problem lies in that there is no free ports to plug the new T1 into on the 2600, so the new T1 is currently going into a secondary and completely separate Netopia router box. Thus, the perceived problem. The current and new T1's are seperate, and I do not see an easy method to bridge them ("easy" would be a multi wan port router that can support all 4 -- 3 are bonded and i believe have to work together, and the new one).

All WAN links are going in to the router, then from the router to various switches/secondary routers, then to the servers. I thought about the idea of placing the new IP's on each server, but presumably I could not use the old IP's on the new T1 and vice versa since the IP addresses are owned by the upper tier ISP's, not us directly which is why I don't think it will work this way. Hence the root of the problem, or what i see as a problem at least.

ok, so both providers give you a local subnet of public addresses? i guess the level you are looking at, you do have a lot of options. I would say put each server on both ip addresses and ave both subnets actually running in a complete state, just leave the default gateway on each server as the current one. at your own leisure you can then change a DNS record and the default gateway of that related bit of tin and migrate it. without firewalling, if old traffic is still hitting the old ip address, then routing it back through the new link mightn't cause any issues anyway... this is clearly something you can test in reverse by connecting to a server on the new ip address before updating dns.

now i have a better feel for your topology this does seem a bit similar to your original suggestion. i guess you were looking at keeping the two networks utterly seperate though?

Well, trying to avoid the cost of buying equipment to "bind" them... and the old service is being shut down on the 23rd, so that equipment will only be a temporary patch anyways. So utterly separate is more by necessity than desire. And yes, both providers have given us separate subnets.

So, I guess my main questions that will determine what path I take are:

1) With the TTL option do I have to do this for every domain we host?
2) If I use NAT, and I DNAT at the old IP, do I have to SNAT at the new IP? Will this affect all natural traffic to the new IP at all? or is there a way to tell if the packet is coming from, or through, the old IP so it doesn't affect the natural traffic to the new IP?

Of coarse this is assuming I can not find a way to set it up on the current setup.

Thanks again

well for a crude temporary solution, you can easily just connect the seperate routers to the same layer 2 infrastructure, with seperate IP ranges on the same nic per server.

That won't work as the IP's are owned by different companies... and correct me if I am wrong, but it would not work to use IP's assigned to different T1 circuit on a different one, right?

For example, we have xyz IP's from Company A, and abc IP's from Company B.

It is not possible to use the IP's from Company A on the circuit from Company B, correct? Or is it? If it is, that would be a simple, yet effective fix for this problem. I am under the assumption that this is not possible.

not on the circuit, but each router connects onwards to a switch right? you can connect both routers to the same switch on different subnets without *your* part of the network.

Although I don't work for an ISP, I am currently transitioning our company to use a new ISP. Almost done!

If I understood acid_kewpie last post correctly, I implemented the same thing he referred to. I put both perimeter routers in the same vlan, so as I transition each device (application/server/firewall), I just change the IP/netmask/gateway of the device. Then that device will "arp" for the new perimeter router within its own broadcast domain. Obviously, I have to deal with DNS changes prior to the ip change of the server/firewall, but I ran down the TTL weeks ago to 2 minutes to reduce the impact of the change.

To illustrate what I have implemented to transition our company to a new ISP, I put together a quick visio drawing (see link below). NOTE: To keep the drawing simple and to the point, I am not showing any firewalls (which is what I am actually changing when I transition to new ISP) or internetworking devices. But they all exist in the same vlan. I am just showing servers with public IP's. Two are using original ISP and two are using new ISP. Make the necessary adjustments in my example to fit your requirements.

See this link: http://www.infohiiway.com/isp-change/example.jpg

Hope this helps! and good luck
--Steve Cowles

I think I understand...Right now the new T1 is sitting all by its lonesome...but you are saying that if I connect it to the same switch the current T1 is plugged into, I should be able to 'use' an ip from either subnet as long as I put in the correct Gateway?