I'm looking for a way (kernel patches, configuration, etc) to bond multiple network interfaces together ... but for limited purposes.

Here's the setup. Machines A, B, C, and D each have 4 NICs, each of which are on separate unmanaged switches. The connections are made in a corresponding way ... e.g. eth0 of each machine are connected via switch 0, eth1 are connected via switch 1, etc. There are also other machines which have only one NIC and are connected to switch 0 only. All NICs for A/B/C/D and the switches are gigabit speed. The remaining machines have a low traffic level. Machines A/B/C/D need the extended bandwidth. And this bandwidth need usually involves only one connection at a time. E.g. machine A transferring files to machine C with no other traffic going on. The speed need is to cut the transfer times from several hours to few hours (such as 8 hours to 2 hours). Transfers of up to a few terabytes at a time are involved.

IEEE 802.1AX won't accomplish this. It requires special support from a single switch that all connection go to (raising costs and reducing reliability). Also, from technical details of 802.1AX, it appears that a decision process is made for which traffic goes over which physical link based on destination information. It's unclear what impact this will have, but it looks like at least a single TCP connection cannot use all physical links, and possbly all traffic from host A to host B is limited to a physical link (not any better than a round robin of crossover cables).

What I am looking for is something that works entirely on an end-to-end basis within a LAN. If it works at the link layer, that could be OK as long as it doesn't have the limitations of 802.1AX. Working at the IP layer would be OK, too (as I can already envision the logic of how to make that work).

This might be an experimental patch to the Linux kernel if anyone has tried it. I have not dug into kernel source to see what might be in there, yet, but will eventually do that if there isn't a patch already available.

Since the switch that eth0 is connected to is unique (in that machine with single nics attach to it) I'd leave eth0 out of the bond and just bond eth1-eth3 on each of the machines with 4 ethernets using the normal bonding module in the kernel using round robin mode (which is the default). Since each of the eth1 through eth3 switches are different they shouldn't be confused by the bonding and don't need to know about it or have special trunking. You'll end up with two different connections to those computers but one will be a 3x capacity and linux can handle potential out of order packets arrivals for a tcp connection

Just wanted to clarify the reason for leaving out eth0, if you left it in the bond then traffic heading to the single nic machines would be really slow since 3/4 of the packets would be lost since they'd go down eth1-eth3 and their switches won't know about the single nic machine

Sorry for the belated followup on this.

What I am thinking about is developing a new kind of bonding/aggregating protocol that fixes these problems. You are pointing one such problem. What needs to be done in this case is each machine needs a kind of "reachability table", perhaps a variation of the ARP table, that tracks for each IP address to be reached, a list of known paths it can be reached via. Imagine the ARP table (indexed by IP address), and yielding an array of {if,mac} tuples. When a packet is to be sent to that IP address, it would be queued in such a way that when any interface in that list is ready to send this packet, it would grab it from the queue (atomically, so not 2 or more grab it), frame it for that mac address, and send it over that interface. Machines connected only on ONE of the many parallel LANs would have just one such entry, and that would operate the usual way (when that interface can get around to transmitting it, it will do so, and not send it at all on a LAN that it can't be received on).

This table would need to be dynamic like the ARP table is now. And it might not be that hard to do because I'm already seeing now that when an ARP request for a given IP is sent, replies come back from multiple interfaces (e.g. if the peer has 6 NICs, I get 6 ARP replies, one for each NIC MAC address ... but only the first ends up in the table as it is now). The ARP query is a broadcast, so that peer machine with 6 NICs is getting it coming in on all 6 NICs, and answering each one with that NIC's MAC. Now I just need to extend this a bit to determine, for N NICs on machine A, and M NICs on machine B, which entries in the NxM matrix to be set true for reachability.

Maybe what I'm really talking about here is a new or modified version of ARP or NDP (for IPv6).

That sounds like standard routing protocols like bgp and ospf with the multiple paths having the same weight so it round robins between them. Don't know if you'd need to make a new bonding protocol maybe just a tweak to a routing protocol/discovery combined with iproute2 routing tables.

However, this would be at the link layer. The only "routing" aspect to this would be to route to the link itself, not to a real router.

ARP looks similar to routing, too. To send to a given IP address, you really send to a given MAC address, based on the table lookup.

Given that what I am pursuing needs to handle both cases where interfaces are on the same LAN or different LANs ... consider eth0 and eth1 on LAN0 and eth2 and eth3 on LAN2, and the peer connected likewise, I'd want to use all 4 NICs. It doesn't matter how the connectivity is structured, I want to use all paths that can reach the peer.

One complication is this. Consider the above scenario. That creates 8 paths: eth0->peer:eth0, eth0->peer:eth1, eth1->peer:eth0 eth1->peer:eth1, eth2->peer:eth2, eth2->peer:eth3, eth3->peer:eth2, eth3->peer:eth3. But I don't see any problem if it just tries to balance across all 8. If there was just one LAN, then we'd have 16 possible paths. Where the complication can have an impact is when there is some assymetry.

If machine A has 4 100mb ethernet NICs and machine B has 1 1000mb ethernet NIC, there are still 4 paths, though clearly all on one LAN.

The approach I see as workable is to enter each packet onto ALL queues at the same time (or some means to effectively do so), such that the first interface to become ready to transmit this packet will get it. The logic of taking the packet out of the queue to send it would need to remove it from ALL queues (so a duplicate is not sent, and so there is no double freeing).

If there was one big "packets to be sent" queue, with each entry marked as to which paths it can be sent over, that could be simpler. When an interface is ready, it grabs the next packet which has a flag for it, and proceeds. But the scenario of sending out via one interface to 4 different interfaces on the peer still needs to be round-robin. But it will be harder to detect when the peer has an overload on just one of those 4 interfaces, for example. Some kind of link layer congestion control seems needed.