Hi folks. I want to do a quick sanity check. I'm working on a project with one of our engineers regarding failover between 2 separate ISPs. We have a router and a PIX firewall incoming from each ISP. This then feeds into a single linux router running zebra that feeds the different networks internally. The dynamic routing is via OSPF for it's load balancing and fault tolerance.

I know we can do load balancing and fault tolerance between the linux router and the 2 firewalls, but my engineer is firm that we can use OSPF to get failure notifications from our 2 ISPs routers as well. I'm almost certain you can't do this especially between 2 different ISPs, but my engineer is boardline genius when it comes to this stuff. I don't want to speak up too loudly if I'm wrong. Am I wrong here or what?

TIA!

why don't you just try it and see.

You need to have your routers, and those of the 2 ISP's be in the same OSPF area.

The desire for fault tolerance between 2 ISPs should tip you off that this is a critical communications line. Usually we do simulate new installs first but I can't simulate ISP side routers I know nothing about.

My line of thinking is that between two ISPs that would not be possible.

Does BGP do fault tolerance and load balancing?

Actually, now that I think about it, if the ISP's don't coordinate your address block via BGP, you're pretty much out of luck. Best way is for you to have your own ASN (and resulting portable IP block) and run BGP at your site. Then work with the ISPs to listen to your BGP...

OSPF can be used to multi-home with the same providers network easily (from multi-site setups, from single-site you can use multi-link PPP), but if you need REAL IP addresses and multihome across ISP's, BGP becomes the tool you should be using. Prerequisite for using BGP is having an ASN and portable IP block assigned to you.

Yes this will work to a certain extent. Your Edge routers need to support OSPF. They will have a default route pointing to the ISP which can be redistributed into OSPF. The Firewalls and the internal router also need to run OSPF. They do not have to be inthe same area but as this is such a tiddly network its pointless to use multiple area's as it just adds complexity and more things to go wrong.

The redistributed default route will now be flooded to the firewalls and internal router.]

Now you can't do per packet load balancing as each packet will get a different source address as it leaves the NAT'd interface of the firewall. If you can configure the internal router to do per-session load balancing then that will work nicely as each session then gets one NAT address or the other, which is fine.

Alternatively if you redistribute the default route with a higher metric on one edge router then you can have a primary and secondary internet access, rather than a load-balanced pair.

If you wanted the same Public address range to be presented to the internet from two different ISPs, thats more complicated. You can do it but only one would be used at a time as BGP does not load balance, there is only one best path. Also the routing you have to the internet at present will most likely be done by static routes on the WAN link. There is no mechanism for withdrawing an invalid route, so you would need to run a routing protocol to your ISPs. This gets very complex and your ISP is not likely to go down that route without you paying them to set it up.

I would just go for per-session load balancing between the two internet public ranges, its nice and simple and you can control it without bothering your ISP.

Thanks! That helps me. My engineer and I will look into something else.

Thanks! I'm sure this would work, but the situation my engineer is talking about is slightly different. He wants OSPF updates from the ISP routers. I guess the end difference would be minimal between this what he's asking. We will discuss this as another option. Thanks again.

Well you need to ask what you are protecting yourself against. The primary risk you should be interested in is WAN circuit failure, edge router failure or pop router failure. Once you are into your ISPs network they should, hopefully, be running any redundancy required to protect you against loss of your internet.

If you run OSPF from the edge router as suggested then if the edge router goes down the route gets withdrawn from your internal router and you reconverge on the other link. If your WAN goes down the route gets withdrawn as well, if the pop router goes down then the WAN goes down and the route gets withdawn and you reconverge.

The only thing this would not protect you against is if the path to the internet fails within the ISP, there is no mechanism for them to tell you the route is unavailable. But as mentioned above this should be unlikely if your ISP is doing his job properly.

Redundancy of any kind is a cost benefit balance like anything else and also suffers from the law of diminishing returns. Doing small amount of reconfig work should cost you little and yet address your most likely risks. Protecting against ALL risks wil cost you lots of cash.But thats your call.

It has been interesting to see the number of incidents where ISP's suffered major network failures in the last couple of years. Not to mention when the big boys start fighting over peering agreements and disconnect from each other... I think that if you have a truly critical site, multiple ISP's are a requirement. However, that really doesn't address ddos protection, etc.

You're not wrong although didn't want to get into that as there's nothing you can do about it anyway except move ISP if yours lets you down. Either that or vet their network very carefully before signing up.

If the network is really critical then you should defintely dual home and preferably to Tier 1 carriers.

Pete, help me out a little. I'm thinking the most likely fault scenario would be a disconnect between my router and the ISP router. How can I have the internal facing interfaces update their OSPF routing tables if the ISP facing interface is not running OSPF?

Its done by redistribution. A static route pointing at the wan interface can then be redistributed into ospf. So the internet router will pass the static route information to its OSPF neighbours. If the physical interface that the static route points at goes down then the static route is no longer valid and will be withdrawn from the route table, and therefore withdrawn from the OSPF database. Your internal router will see the OSPF database change as it is propogated by your firewalls. It now knows that the default route via the internet router is no longer good and will now route all traffic to the other internet router which is still advertising a valid route.

If your internet router is a cisco I can jot you down a sample config if you want.

They are Ciscos so absolutely! Thanks!

I guess my engineer was mostly correct then. Dang! I usually use Cisco end-to-end, so I use EIGRP 99% of the time. My OSPF skills are a little rusty. I use Linux for a lot of stuff, but he uses it for almost everything including routing. That's fine, but Linux can't do EIGRP. I'm getting the hang of it though!

Thanks again guys! That sample config should help me.