Bonded interface failover speed question

parthelion · 07-10-2013, 05:37 PM

Hi everyone,

I'm having a bit of a problem with a bond interface. Essentially, my configuration is the following:

Code:


  Box1
    |
  Switch
  |    |
 Box2  Box3
    \  /
    Box4

Boxes 1,2,3 and 4 are all RHEL5 servers. The critical communications path I'm concerned with is between Box1 and Box4. Box4 cannot be directly connected to the switch, so traffic is routed through box2 and box3. I have to handle losing either of those two boxes. I'm accomplishing this by bonding box4's interfaces to boxes 2 and 3, and having box1 toggle its route to box4 between the two intermediate boxes if it loses its connection.

This all seems to work pretty well, but my problem is that if box4's bonded interface is talking through box2 and I sever that interface by any means, it seems to take about 5 seconds before communications between box3 and box4 come up (and vice-versa.) I'd like this to be faster. Checking the active-slave file for the bonded interface shows me that the bonded interface seems to swap over almost instantly. Using ifconfig also shows that box4 immediately starts trying to use the correct new interface, but it's still about 5 seconds before I can ping between box4 and the new intermediate box. Does anyone have any ideas as to how I might improve this time? Or am I going about this completely the wrong way?

This is box4's modprobe.conf:

Code:

alias bond0 bonding
options bonding miimon=100 mode=1

Its ifcfg-bond0:

Code:

DEVICE=bond0
IPADDR=10.7.12.4
NETWORK=10.7.12.0
NETMASK=255.255.255.0
GATEWAY=10.7.12.1
BROADCAST=10.7.12.255
USERCTL=no
BOOTPROTO=none
ONBOOT=yes

And the two slave interfaces:

Code:

DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes

Code:

DEVICE=eth1
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes

The two box4-facing interfaces are nothing special (and both have the IP 10.7.12.3).

Thanks for your help, everyone. I've been banging my head against the wall for a couple days.

mpapet · 07-10-2013, 11:23 PM

IMHO, the better solution is to cluster a virtual IP interface between the two boxes. When one drops, the other takes over. For me, the takeover is almost instantaneous. It also handles fluttering up/down/up/down situations well. RHEL5's cluster is solid, but crusty code now. It's got a gui too, so not too much meddling about for a simple failover NIC.

Another alternative is to use SNMP to trigger a script to clean up after the downed interface. That's pretty scripty/SNMP mucking about, but might be a less intimidating learning curve than a heartbeat cluster.

I'd be interested to see the kernel messages when one of the bonded interfaces fail. Do you know what causes the delay?