Hi there LQ peeps,
I just setup a new LAMP server (CentOS 5.5 x86_64) box with channel bonding on NetXtreme II BCM5709 Gigabit Ethernet (IBM x3650 M3). The problem is I wasn't able to connect to this server when I'm in different VLAN's. This server also unable to ping different VLAN's. But everything works fine when I transact in the same VLAN.

Here's the config:

Datacenter VLAN = 9

Firewall = Disabled
SELinux = Disabled

bond0

DEVICE=bond0
IPADDR=10.238.9.254
NETMASK=255.255.255.0
GATEWAY=10.238.9.1
ONBOOT=yes
USERCTL=no
BONDING_OPTS="mode=1 miimon=100"

eth0

# Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet
# Bonding Configuration
DEVICE=eth0
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no

eth1

# Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet
# Bonding Configuration
DEVICE=eth1
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no

Need help please. Thanks.

OK, there is no complete picture here at all. You've provided a vlan id, yet there is no mention at all about vlans in the rest of the config. what's that about? vlans appear to be utterly irrelevant here, and it's just a bonding issue. Also bonding itself looks irrelevant if you've got a mode 1 working fine. So basic troubleshooting to reach a different subnet says 1) can you ping the default gateway and does it show up in your arp cache. 2) what does your routing table actually say? (route -n) also it may be worth showing us a full ifconfig output as well as "cat /proc/net/bonding/bond0" something on the bonding is going to be wrong (unless you pasted the wrong info) as both eth files say they are eth0...

Hi Chris,
Sorry for the incomplete info. Anyways here are some additional info.

[root@webrwm02 ~]# ping 10.238.9.1
PING 10.238.9.1 (10.238.9.1) 56(84) bytes of data.
64 bytes from 10.238.9.1: icmp_seq=1 ttl=255 time=2.18 ms
64 bytes from 10.238.9.1: icmp_seq=2 ttl=255 time=1.62 ms
64 bytes from 10.238.9.1: icmp_seq=3 ttl=255 time=5.31 ms

--- 10.238.9.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 1.628/3.043/5.313/1.621 ms

[root@webrwm02 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.238.9.0 0.0.0.0 255.255.255.0 U 0 0 0 bond0
10.238.9.0 0.0.0.0 255.255.255.0 U 1 0 0 eth1
10.238.9.0 0.0.0.0 255.255.255.0 U 1 0 0 eth0
169.254.95.0 0.0.0.0 255.255.255.0 U 0 0 0 usb0
0.0.0.0 10.238.9.1 0.0.0.0 UG 0 0 0 eth0

[root@webrwm02 ~]# ifconfig
bond0 Link encap:Ethernet HWaddr E4:1F:13:B6:31:C0
inet addr:10.238.9.254 Bcast:10.238.9.255 Mask:255.255.255.0
inet6 addr: fe80::e61f:13ff:feb6:31c0/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:42224 errors:0 dropped:0 overruns:0 frame:0
TX packets:5902 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:35313551 (33.6 MiB) TX bytes:455066 (444.4 KiB)

eth0 Link encap:Ethernet HWaddr E4:1F:13:B6:31:C0
inet addr:10.238.9.254 Bcast:10.238.9.255 Mask:255.255.255.0
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:23417 errors:0 dropped:0 overruns:0 frame:0
TX packets:5874 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:17838592 (17.0 MiB) TX bytes:451582 (440.9 KiB)
Interrupt:169 Memory:96000000-96012800

eth1 Link encap:Ethernet HWaddr E4:1F:13:B6:31:C0
inet addr:10.238.9.254 Bcast:10.238.9.255 Mask:255.255.255.0
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:18807 errors:0 dropped:0 overruns:0 frame:0
TX packets:28 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:17474959 (16.6 MiB) TX bytes:3484 (3.4 KiB)
Interrupt:138 Memory:98000000-98012800

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:2664 errors:0 dropped:0 overruns:0 frame:0
TX packets:2664 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1992858 (1.9 MiB) TX bytes:1992858 (1.9 MiB)

usb0 Link encap:Ethernet HWaddr E6:1F:13:A9:31:C3
inet addr:169.254.95.120 Bcast:169.254.95.255 Mask:255.255.255.0
inet6 addr: fe80::e41f:13ff:fea9:31c3/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4029 errors:0 dropped:0 overruns:0 frame:0
TX packets:1851 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:271174 (264.8 KiB) TX bytes:171887 (167.8 KiB)

[root@webrwm02 ~]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.4.0 (October 7, 2008)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: e4:1f:13:b6:31:c0

Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: e4:1f:13:b6:31:c2

Also tried mode=5 and 6 but still no luck. Thanks.

OK, so where is the ping that's not working? have you verified that eth0 configure without a bond works in the same way? As you're using mode one, bond0 should just behave exactly the same as eth0, as eth1 is going to be doing absolutely nothing.

Hi Chris,
Here please have a look. Thanks.

[root@webrwm02 ~]# ping 10.238.150.1
PING 10.238.150.1 (10.238.150.1) 56(84) bytes of data.
From 10.238.9.254 icmp_seq=2 Destination Host Unreachable
From 10.238.9.254 icmp_seq=3 Destination Host Unreachable
From 10.238.9.254 icmp_seq=4 Destination Host Unreachable

--- 10.238.150.1 ping statistics ---
5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3999ms
, pipe 3

Hmm, your default gateway is set on eth0, not bond0, change that and you should be OK. I'm *pretty* sure that the route will always bo on the bonded interface, not the active nic, doesn't make sense any other way. Your config files read ok though, which seems odd.

Hi Chris,
As you can see my gateway already defined in bond0, please correct me if I'm wrong. Thank you very much.

bond0

DEVICE=bond0
IPADDR=10.238.9.254
NETMASK=255.255.255.0
GATEWAY=10.238.9.1
ONBOOT=yes
USERCTL=no
BONDING_OPTS="mode=1 miimon=100"

yes, it is defined there, so is a little odd to me. If you remove the droute and add it manually to bond0 - "route del default gw 10.238.9.1" and then add it "route add default gw 10.238.9.1 dev bond0" it should add it against the bond0 interface, which makes sense from the bonding side of things.

Hi Chris,
Tried these command:
route del default gw 10.238.9.1
route add default gw 10.238.9.1 dev bond0

But still no luck. Any idea?

well does this actually work without the bond?

Hi Chris,
Yes it definitely works without bonding but since its a web server it is required to configured the bonding option to provide load balancing and redundancy. Please advice. Thank you.