bonding 802.3ad problem
 

Hello,

I am bonding 3 interfaces together using mode=4 (802.3ad link aggregation) but having problems. When I reboot, the bond0 interface will come up but no activity. Once I get logged in, I do a 'service network restart' and things come back fine. First, here's my config. Errors at the bottom.

[root@omadvnfs01c ~]# cat /etc/modprobe.conf
alias eth0 bnx2
alias eth1 bnx2
alias scsi_hostadapter megaraid_sas
alias eth2 igb
alias eth3 igb
alias eth4 igb
alias eth5 igb
alias bond0 bonding
options bond0 miimon=80 mode=4
[root@omadvnfs01c ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond0
GATEWAY=10.198.1.1
DEVICE=bond0
BOOTPROTO=none
NETMASK=255.255.255.0
IPADDR=10.198.1.109
ONBOOT=yes
USERCTL=no
MTU=9000
[root@omadvnfs01c ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
# Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet
DEVICE=eth0
BOOTPROTO=none
HWADDR=00:13:72:FA:DB:A3
ONBOOT=yes
SLAVE=yes
MASTER=bond0
USERCTL=no
MTU=9000
[root@omadvnfs01c ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
# Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet
DEVICE=eth1
BOOTPROTO=none
HWADDR=00:13:72:FA:DB:A5
ONBOOT=yes
SLAVE=yes
MASTER=bond0
USERCTL=no
MTU=9000
[root@omadvnfs01c ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth3
# Intel Corporation 82575GB Gigabit Network Connection
DEVICE=eth3
BOOTPROTO=none
ONBOOT=yes
HWADDR=00:1b:21:3d:e1:91
SLAVE=yes
MASTER=bond0
USERCTL=no
MTU=9000


Lots of errors.

[root@omadvnfs01c ~]# dmesg | grep bond0
ADDRCONF(NETDEV_UP): bond0: link is not ready
bonding: bond0: Adding slave eth0.
bonding: bond0: enslaving eth0 as a backup interface with a down link.
bonding: bond0: Adding slave eth1.
bonding: bond0: enslaving eth1 as a backup interface with a down link.
bonding: bond0: link status definitely up for interface eth0.
bonding: bond0: link status definitely up for interface eth1.
bonding: bond0: Adding slave eth3.
bonding: bond0: enslaving eth3 as a backup interface with a down link.
bonding: bond0: link status definitely down for interface eth1, disabling it
bonding: bond0: link status definitely up for interface eth3.
bonding: bond0: link status definitely up for interface eth1.
bonding: bond0: link status definitely down for interface eth3, disabling it
bonding: bond0: link status definitely up for interface eth3.
bonding: bond0: link status definitely down for interface eth0, disabling it
bonding: bond0: link status definitely down for interface eth1, disabling it
ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
bonding: bond0: link status definitely up for interface eth0.
bonding: bond0: link status definitely up for interface eth1.
bond0: no IPv6 routers present
bonding: bond0: Removing slave eth0
bonding: bond0: Warning: the permanent HWaddr of eth0 - 00:13:72:FA:DB:A3 - is still in use by bond0. Set the HWaddr of eth0 to a different address to avoid conflicts.
bonding: bond0: releasing backup interface eth0
bonding: bond0: Removing slave eth1
bonding: bond0: releasing backup interface eth1
bonding: bond0: Removing slave eth3
bonding: bond0: Removing an active aggregator
bonding: bond0: releasing active interface eth3
ADDRCONF(NETDEV_UP): bond0: link is not ready
bonding: bond0: Adding slave eth0.
bonding: bond0: enslaving eth0 as a backup interface with a down link.
bonding: bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond
bonding: bond0: Adding slave eth1.
bonding: bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond
bonding: bond0: enslaving eth1 as a backup interface with a down link.
ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
bonding: bond0: link status definitely up for interface eth0.
bonding: bond0: link status definitely up for interface eth1.
bonding: bond0: Adding slave eth3.
bonding: bond0: enslaving eth3 as a backup interface with a down link.
bonding: bond0: link status definitely down for interface eth1, disabling it
bonding: bond0: link status definitely up for interface eth3.
bonding: bond0: link status definitely down for interface eth3, disabling it
bonding: bond0: link status definitely up for interface eth1.
bonding: bond0: link status definitely up for interface eth3.
bonding: bond0: link status definitely down for interface eth0, disabling it
bonding: bond0: link status definitely down for interface eth1, disabling it
bonding: bond0: link status definitely down for interface eth3, disabling it
bonding: bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond
bonding: bond0: link status definitely up for interface eth3.
bonding: bond0: link status definitely up for interface eth1.
bond0: no IPv6 routers present
bonding: bond0: link status definitely up for interface eth0.

well *when* were those errors? without timestamps it's hard to pu tit into context. Do you have a suitable switch at the other end? You shouldn't have 3 members on a bond, stick to powers of 2 - 2, 4, 8 etc...

The errors are at boot time from dmesg. As soon as I was able to log in, I restarted the network and that's when the link up messages started. I haven't heard the powers of 2 advice yet. What's the logic behind that?

well there are ups and downs and ups again... what was when exactly? Is there enything else than the bond0 entries? What does the switch side logs say?