Stand-by machine in case real machine crashes
I am planning to create a machine to take over the functions of my firewall/router in case the real machine crashes.
The situation is this:
I have two sites, site A and site B. Both site A and site B have an internet connection. At both sites I have a firewall to connect the internal network to the internet. Both firewalls are connected thru a wireless connection using a wireless card in both machines.
At site A I have a total of 3 NIC's in the machine (LAN, internet and wireless). At site B I have 4 NIC's in the machine. (2 LAN's, internet and wireless).
LAN-A and LAN-B are interconnected thru the wireless link. Access is somewhat restricted, that is, not all parts of both LANs are allowed to see each other. Both LAN-A and LAN-B are able to use the internet connection on both sites.
Both firewalls run a traffic shaper, but are configured differently. The firewall at site A runs a DNS server, a DHCP server and a traffic log.
This all sounds complicated, but unfortunately it is necessary. It all works perfect. The point is that both machines are very similar, but yet have a lot of differences in the configuration.
What I now need to do is to be prepared for disaster. My entire company is relying on the wireless connection and the internet connections. I have to be prepared that one firewall will be completely defective , and that it should be replaced by a different machine.
So... what would be the best way to have a stand-by machine prepared and completely updated with the configuration of the real machine? I am sure calling for a disaster if do not provide some update mechanism between the real and the stand-by machine. Configurations tend to drift, and the stand-by machine might remain unused for years (or forever hopefully).
The current idea is this:
I build another machine, and install 3 complete Linux installations on it.
1. as stand-by machine doing nothing
2. as firewall A
3. as firewall B
Each installation would be completely independent.
When booting, you can choose which function the machine will perform. As long as the machine is not needed, it is configured as stand-by, doing nothing and connected to the LAN. This machine can however access the partitions which are used for firewall A and firewall B.
Both firewalls A and B do a daily rsync of the important directories with the corresponding directories on the stand-by machine. As soon as one of the machines fail, I would take the stand-by machine, connect it in place of the defective one and boot for the correct function.
This means that a hot stand-by is not necessary. I always have access to both sites, and can afford to go down to the site and put the replacement there. However, it should be a no-brainer and no configuration should be necessary. This precludes having an "empty machine" on which I should restore a backup before I can use it.
I am not too economical to have 2 machines as spare, but since it should be connected to the network for updates, I think I still need two Linux installations on each machine, one for running as stand-by and receiving updates, and one as replacement for the defective machine. So having 2 machines would only increase my manageability problem.
Any comments or better ideas someone?