I'm setting up a server which has 4 NICs and 2 IP addresses on the same subnet, with the intention of having:

bond0 = system = eth0 + eth2 = 10.80.132.94
bond1 = application = eth1 + eth3 = 10.80.132.97

I've got bond0 set up, and my default route is attached to it:

root# ip route show
10.80.132.0/23 dev bond0 proto kernel scope link src 10.80.132.94
10.80.132.0/23 dev eth3 proto kernel scope link src 10.80.132.97
169.254.0.0/16 dev bond0 scope link
default via 10.80.132.11 dev bond0
root#

I am testing the other two interfaces have connectivity before setting up bond1, but I am having problems...

I am testing by pinging the "97" IP from a server on the same subnet (ie. local) and a server on another subnet (ie. remote).

When I assign the "97" IP to eth1 I get a response to my ping from both local and remote servers.

However, if I swap the IP to eth3 (I assign 0.0.0.0 to eth1 and down it, and then assign the IP to eth3 and up it), it doesn't work properly; I still get a local ping response but I don't get a remote one. (But it may spring to life after perhaps 20 minutes, but then swapping the IP back to eth1 makes it hang.)

I think the problem may be to do with the outbound traffic going over bond0 instead of back over the receiving interface.

I've done a fair amount of searching the internet and I thought it might be the so-called "arp flux" problem, so I tried setting the arp_filter to 1 for each interface, but it doesn't seem to have any effect. (My NICs don't have the "hidden" option there, which is the other thing mentioned in relation to the arp flux.)

My machine is a HP blade running RHEL 5.7, which is patched to a grab of what was available in August 2011.

Thanks for your help!

Its probably reverse path filtering, theres proc entries called rp_filter. Google for a good way to set it for your distro

Running with two physical network links on the same subnet is serious networking pain, you will have numerous issues.

In your situation, why not bond all 4 links together, and use an ip alias instead? This will work way better for you.

Thanks for the reply.

I've set rp_filter on bond0 (which is on eth0 & eth2) and on each of the two interfaces I'm trying to switch between.

But then the remote stops regardless of which interface is active(and there is a gap in my icmp_seq); the local carries on.


FYI, I inherited an environment with UNIX (HP-UX) and Linux servers with multiple NICs on the same subnet, but it seems like I'm being a bit of a maverick in implementing bonding. However, I'm only implementing fault-tolerant (mode 1 = active/standby), as the network infrastructure isn't set up for active/active (LACP). My fallback will be to just use aliases on the primary interfaces pair.

Are you using your ethernet devices in the bond and outside it?

The routing table shows eth3 and not bond1 has been assigned the subnet..

One thing to realise which helps one think about this sort of thing,
is that an interface doesn't really have an ip address, (although its configured like that)
a host has the ip addresses, and an interface has the subnets.

You can use advanced routing stuff to try and get packets with the right source to leave the correct interface, but you will
need to tweak the routing tables, the arp_ignore settings and the rp_filter.

The kernel will arp reply on an ethernet interface for nearly any ip that it has, if there is a query, this can cause weirdness too
so try looking at the arp_ignore settings too.

You _can_ make this setup work, but it is a seriously confusing pain.

No - I haven't set up bond1 yet.

My current setup is 1 bonded pair plus 2 individual NICs that I am manually swapping the second address between:

ie.
bond0 (eth0 + eth2) = xx.xx.xx.94
eth1 = xx.xx.xx.94
eth3 = 0.0.0.0 down

or

bond0 (eth0 + eth2) = xx.xx.xx.94
eth1 = 0.0.0.0 down
eth3 = xx.xx.xx.94

I was testing each interface had connectivity before setting up the bonding.