Hello,

Anyone know how the ALB and TL; modes 5 and 6 work on the bonding driver in linux?

Do they xor the src/dst mac andor ip?

Appears that they just "etherchannel" each route based on the dstip.


for two connections, going from a host a to host b. (using a aliased ip on the bond)

srcip(same)dstip(different) = using both slaves, 2gigabit transfer rates
srcip(same)dstip(same) = using 1 slave, 1 gigabit
srcip(different)dstip(same) = using 1 slave, 1 gigabit


Below are my test result so far:

Hardware

DL585G1 x 2
Cisco 3750

Benchmark Tools

Netcat

Setup a listen server
#nc -l -p 7778 > /dev/null
Listens on port 7778 and transfers the output to /dev/null. Use /dev/null to get rid of the hdd write speed caps.

Transfer a file
#time cat test_500mb | nc -s 192.168.0.101 192.168.0.125 777
This will send the file test_500mb through local interface 192.168.0.101 to 192.168.0.125 using port 7777. Typically transfer from a ramdisk, to avoid hard drive read speed cap.

Benchmarks

Loopback check
235MB/s
1882mbs

Round Robin mode= 0:


Round Robin will (tx)transmit a packet onto each slave, dividing a single connection into multiple streams. The receive side is completely controlled by the switches / router that you are going through. Also round robin can have packet order issues, resulting in slower then expected speeds with more overhead. Round Robin will always (TX)transmit packets across ALL active slave interfaces.

Routing:
Typically switches will route etherchannels based on either {src-mac | dst-mac | src-dst-mac | src-ip | dst-ip | src-dst-ip }. So if your sending 1 connection to multiple destinations you will get 2gigabit with the dst-mac, dst-ip, or either of the xor methods. The src-mac and src-ip will result in only 1gigabit, due to a bond having static mac/ip for both slave devices. Also hitting a gateway or a single host will result in only 1 gigabit (a gateway is considered the same as hitting a single host by the etherchannel). The only way around this is to use a Layer4 switch that allows routing etherchannels by port (src-port | dst-port | src-dst-port). Typically the xor methods allow for a better bandwidth distribution. The xor port method allow for the best distribution for RR a bond.


Examples:

1 host> eth6/eth7 = Round Robin bound> cisco> 1000 hosts=2 gigabit
(xor,dst)

1000 hosts> eth6/eth7 = Round Robin bound> cisco> 1 host=2 gigabit
(xor only!)

(Requires a L4 switch)
1 host> eth6/eth7 = Round Robin bound> cisco(L4)> 1 host=2 gigabit
(src-port | dst-port | src-dst-port only!)

Results:
---------------------------------------------
RR 2 transfers to 2 different ip’s src xor dst ip 2 slaves active:
Transferred 1024MB in 4.558s
224MB/s
1797mbs
---------------------------------------------
RR 2 transfers 1 slave active:
Transferred 1024MB in 8.513s
120MB/s
962mbs
----------------------------------------------



ALB mode = 6:




Matrix of Cisco Load Balancing Methods
This matrix consolidates the load balancing methods that this document describes:
Layer 3 = port-channel load-balance {src-mac | dst-mac | src-dst-mac | src-ip | dst-ip | src-dst-ip | src-port | dst-port | src-dst-port}
Layer 4 = port-channel load-balance {src-mac | dst-mac | src-dst-mac | src-ip | dst-ip | src-dst-ip | src-port | dst-port | src-dst-port | mpls}

Platform Address Used in XOR Source-Based? Destination-Based? Source-Destination-Based? Load Balancing Method—Configurable/Fixed?
6500/6000 Layer 2, Layer 3 addresses, Layer 4 information, or MPLS information2 Yes Yes Yes Configurable
5500/5000 Layer 2 address only — — Yes Cannot change the method
4500/4000 Layer 2, Layer 3 addresses, or Layer 4 information Yes Yes Yes Configurable
2900XL/3500XL Layer 2 address only Yes Yes — Configurable
3750/3560 Layer 2 or Layer 3 address only Yes Yes Yes Configurable
2950/2955/3550 Layer 2 address only1 Yes Yes —1 Configurable
1900/2820 These platforms use a special method of load balancing. See the Catalyst 1900/2820 section for details.

8500 Layer 3 address only — — Yes Cannot change the metho


Enable src-dst-ip on switch, without src-dst-ip enabled the packets will
route based on mac address, basically disabling the bond.


Switch#configure
Configuring from terminal, memory, or network [terminal]?
Enter configuration commands, one per line. End with CNTL/Z.
Switch(config)#port-channel load-balance ?
dst-ip Dst IP Addr
dst-mac Dst Mac Addr
src-dst-ip Src XOR Dst IP Addr
src-dst-mac Src XOR Dst Mac Addr
src-ip Src IP Addr
src-mac Src Mac Addr

Switch(config)#port-channel load-balance src-dst-ip

Switch#show etherchannel load-balance
EtherChannel Load-Balancing Operational State (src-dst-ip):
Non-IP: Source XOR Destination MAC address
IPv4: Source XOR Destination IP address
IPv6: Source XOR Destination IP address


------------------------------------------------------------

Example setup a cisco 3750 for LACP(mode 4 only) or PAGP bonding.

Cisco IOS Configuration
Cisco CatOS based devices require alternate configuration.

We require:

1. A port-channel interface to represent the bonded/teamed interfaces
2. Physical interfaces to join the port-channel
3. The configurations below add trunking information

note: for PAGP(mode=0,2,3) subistutue the following settings below
channel-protocol pagp
channel-group 1 mode auto


-
configure terminal
interface Port-channel1
description LACP Channel Bundle for virt-host-1
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan all
switchport trunk native vlan 1
switchport mode trunk
no ip address
no shutdown
exit

Add physical interfaces to the bonded interface (max 8), note that the
"switchport" instructions must match.
-
!
configure terminal
interface GigabitEthernet1/0/1
description virt-host-1 port 1 (eth0)
switchport
switchport trunk encapsulation dot1q <<< sets trunk type, dot1q vs Cisco ISL
switchport trunk allowed vlan all <<< select vlans to expose on trunk
switchport trunk native vlan 1 <<< forces the native VLAN (see warning)
switchport mode trunk <<< puts the interface into trunk (802.1q) mode
no ip address
no shutdown
channel-protocol lacp <<< selects lacp as the bonding protocol (802.3ad)
channel-group 1 mode active <<< channel-group # must match port-channel #
!
configure terminal
interface GigabitEthernet1/0/2
description virt-host-1 port 2 (eth1)
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 1,50-89
switchport trunk native vlan 1
switchport mode trunk
no ip address
no shutdown
channel-protocol lacp
channel-group 1 mode active
!