LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices

Reply
 
Search this Thread
Old 04-14-2011, 01:52 AM   #1
td3201
Member
 
Registered: Jan 2002
Location: Omaha, NE US
Distribution: Red Hat/CentOS
Posts: 224

Rep: Reputation: 30
802.3ad bonding not using all interfaces equally


Hello,

I am seeing an issue on a few servers where it doesn't appear all NICs in the 802.3ad lag are all operating at the same level. A few of the servers have two bonds each with two NICs in each bond. I have two NFS servers that each has 1 bond with 3 NICs. All are RHEL5 x64 2.6.18.

Here's one of my bonds:
[root@server ~]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.4.0-1 (October 7, 2008)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 80
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Active Aggregator Info:
Aggregator ID: 3
Number of ports: 3
Actor Key: 17
Partner Key: 10010
Partner Mac Address: 00:1b:ed:80:17:c0

Slave Interface: eth0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 2
Permanent HW addr: 00:1d:09:71:97:3f
Aggregator ID: 3

Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 3
Permanent HW addr: 00:1d:09:71:97:41
Aggregator ID: 3

Slave Interface: eth3
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 2
Permanent HW addr: 00:15:17:6c:e4:89
Aggregator ID: 3


Here's a snippet of sar:
[root@server ~]# sar -n DEV 1 5

Average: IFACE rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/s rxmcst/s
Average: lo 0.40 0.40 33.67 33.67 0.00 0.00 0.00
Average: eth0 1924.25 0.00 1574977.35 0.00 0.00 0.00 0.00
Average: eth1 7034.87 524.25 41138060.92 3758143.29 0.00 0.00 0.00
Average: eth2 4608.82 3578.56 11813732.26 17426152.71 0.00 0.00 0.00
Average: eth3 352.91 7568.94 174972.34 31082267.33 0.00 0.00 0.00
Average: eth4 4613.83 3667.94 12243683.77 17352368.34 0.00 0.00 0.00
Average: eth5 4533.87 3526.25 11913877.35 17389009.22 0.00 0.00 0.00
Average: sit0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: bond0 9312.02 8093.19 42888010.62 34840410.62 0.00 0.00 0.00

I couldn't read that without having a stroke so here it is broken down in MB/s by interface:
rx MB/s tx MB/s
eth0 1.5 0.0
eth1 39.2 3.6
eth3 0.2 29.6
bond0 40.9 33.2

I think the reason why I see one interface dominating RX and another dominating TX is due to the xmit_hash_policy but there are three hosts that use this particular server for network traffic. That's 3 different physical mac addresses. The layer2 algorithm should be fine in that situation I would think. What am I missing? Would I just be better off with balance-rr?

Thanks!
 
Old 04-14-2011, 07:07 PM   #2
kbp
Senior Member
 
Registered: Aug 2009
Posts: 3,758

Rep: Reputation: 644Reputation: 644Reputation: 644Reputation: 644Reputation: 644Reputation: 644
You could try balance-alb (mode 6), this will enable receive load balancing as well. Try reading bonding.txt and see what suits your traffic patterns

hth
 
Old 04-15-2011, 05:53 PM   #3
td3201
Member
 
Registered: Jan 2002
Location: Omaha, NE US
Distribution: Red Hat/CentOS
Posts: 224

Original Poster
Rep: Reputation: 30
This is expected. Need an even number of interfaces. I saw a reasonable even distribution with two interfaces.
 
Old 04-21-2011, 10:04 PM   #4
td3201
Member
 
Registered: Jan 2002
Location: Omaha, NE US
Distribution: Red Hat/CentOS
Posts: 224

Original Poster
Rep: Reputation: 30
It appears mode 6 does things more evenly across 3 interfaces. If I ever go to 4, I will try 802.3ad again.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Bonding a Bond running 802.3ad Dynamic link aggregation kas22 Linux - Networking 4 10-20-2011 05:24 PM
802.3ad Performance TruckStuff Linux - Networking 1 04-01-2010 01:55 PM
bonding 802.3ad problem td3201 Linux - Networking 3 12-18-2009 01:04 PM
URGENT - Not able to bring up 802.3ad bonding kanth001 Linux - Networking 1 12-18-2009 10:31 AM
bonding 802.3ad -- sanity check td3201 Linux - Networking 1 04-07-2009 12:50 PM


All times are GMT -5. The time now is 12:31 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration