LinuxQuestions.org - 2 ISPs with auto-switching when 1 is down.

RFC: Request for Comments

My company has two static ADSLs from different ISP serve as load balancing and backup. Any better ideas, suggestions, corrections are most welcome. I am not an expert in Linux or Networking, so please feel free to correct me if any part is not right.

My plan was almost perfect, except for the DNS zone transfer.

I have searched the forum and there is no concreate answers. Maybe those who are interest in this topic can join in as well. Please read on, a little long post.

My current setup:
Internet -> ISP1 Router -> Switch(Ext) -> FW -> Switch(Int) -> Local LAN
Internet -> ISP2 Router -> Switch(Ext) -> FW -> Switch(Int) -> Local LAN

Switch (Ext) <- (DMZ) Few Public Servers (Mail & WWW)

All public servers have 2 NICs (ISP1, ISP2), FW had 3 NICs (Local, ISP1, ISP2).

My logic for load balance and backup.
Incoming traffic from ISP1, Outgoing traffic to ISP2.
Public servers will ping ISP1 and ISP2 remote WAN at a fixed interval.
If ISP1 down - change default route to ISP2
If ISP2 down - change default route to ISP1

my link-chk script
***************
isp1_packet_count=`ping ISP1_WAN_IP -c 5 |grep trans | awk '{ print substr($4,1,1) }'`
isp2_packet_count=`ping ISP2_WAN_IP -c 5 |grep trans | awk '{ print substr($4,1,1) }'`

# If packet drop more than 20%, change link
if [ "$isp1_packet_count" -ge "4" ]
then
echo "***************************************** "
echo " ** NO CHANGES NEEDED ** "
echo "***************************************** "
else
echo "***************************************** "
echo " ** NOTICE : Link quality drop. ** "
echo "***************************************** "
/usr/local/bin/dns-isp2
killall named;named -u nobody
route add default gw ISP2_Router_IP metric 0 eth2
fi
# If packet drop more than 20%, change link
if [ "$isp2_packet_count" -ge "4" ]
then
echo "***************************************** "
echo " ** NO CHANGES NEEDED ** "
echo "***************************************** "
else
echo "***************************************** "
echo " ** NOTICE : Link quality drop. ** "
echo "***************************************** "
/usr/local/bin/dns-isp1
killall named;named -u nobody
route del default gw ISP1_Router_IP metric 0 eth2
fi

** END SCRIPT **

My public servers' IPs are with ISP1. (eg. www.domain.com -> ISP1.x.x.x)
When ISP1 down my DNS will copy the zone files with ISP2 IPs and restart. (dns-isp1 and dns-isp2) (eg. www.domain.com -> ISP2.x.x.x)

All is well here except for my secondary DNS (two of them).
My script will run at the fixed interval and remove the zonefiles and restart the DNS server.

I have the similar link-chk script above with the exception of the commands to be run when failed.

If ISP1 failed
rm -f /var/named/*.hosts
cp -p /etc/named.conf.isp2 /etc/named.conf
killall named;named -u nobody

If ISP2 failed
rm -f /var/named/*.hosts
cp -p /etc/named.conf.isp1 /etc/named.conf
killall named;named -u nobody

In this script, my secondary will keep deleting the zone files if one of the link is still down. I was thinking of manual intervention here. Anyone have any idea, how can I make this link-chk script handle the situation?

Sorry if this post is a little long winded.... Hope to have a good discussion on this topic.