LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices

Reply
 
Search this Thread
Old 12-07-2008, 03:11 PM   #1
zerobane
Member
 
Registered: Jan 2006
Posts: 41

Rep: Reputation: 16
redhat el4 bonding round robin and LACP testin, bad results help?


Hello,

Attempting to bond two hosts through a cisco 3750 and then do a transfer test. I wanted to see the differnces and caveats of mode 0 round robin vs mode 4 LACP and modes 5 TLB and 6 ALB.

I've tried both and my results are identical to a single NIC no bond.

I;ve been using nc (netcat to test)
time cat test_500mb | nc 192.168.0.125 7778 #on host 1
nc -l -p 7778 > /dev/null #on host 2


Test results:
-----------------------------------------------------------------
Loopback to null sanity test:
235MB/s
1882mbs


------------------------------------------------
LACP is routed by mac addresses, so a point to point transfer test or using

a gateway will only use 1 slave device!!!

LACP 1 transfer 2 slaves active:
112MB/s
900mbs

LACP 1 transfer 1 slave active:
112MB/s
900mbs


------------------------------------------------

LACP 2 transfers 2 slaves active:
112MB/s
900mbs

LACP 2 transfers 1 slave active:
112MB/s
898mbs

-------------------------------------------------
Round Robin 2 transfers 2 slaves active:
112MB/s
900mbs

Round Robin 2 transfers 1 slave active:
112MB/s
898mbs


Hosts Config.

[root@nps26006 ~]# cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v2.6.1 (October 29, 2004)

Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth5
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:10:18:2c:b6:54

Slave Interface: eth6
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:10:18:2c:b6:55



-----------------------------------------------------------

cisco config

Switch#show running-config
Building configuration...

Current configuration : 2075 bytes
!
version 12.2
no service pad
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
!
hostname Switch
!

!
no aaa new-model
switch 1 provision ws-c3750g-24t
vtp domain ciscovtp
vtp mode transparent
ip subnet-zero
!
!
!
!
port-channel load-balance src-dst-mac
no file verify auto
spanning-tree mode pvst
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
interface Port-channel1
switchport trunk allowed vlan 1-3
!
interface Port-channel2
no switchport
no ip address
!
interface Port-channel3
no switchport
no ip address
!
interface Port-channel4
switchport trunk allowed vlan 1-3
!
interface Port-channel5
!
interface Port-channel6
!
interface GigabitEthernet1/0/1
switchport trunk allowed vlan 1-3
channel-group 5 mode auto
!
interface GigabitEthernet1/0/2
switchport trunk allowed vlan 1-3
channel-group 5 mode auto
!
interface GigabitEthernet1/0/3
!
interface GigabitEthernet1/0/4
!
interface GigabitEthernet1/0/5
!
interface GigabitEthernet1/0/6
!
interface GigabitEthernet1/0/7
!
interface GigabitEthernet1/0/8
!
interface GigabitEthernet1/0/9
!
interface GigabitEthernet1/0/10
!
interface GigabitEthernet1/0/11
!
interface GigabitEthernet1/0/12
!
interface GigabitEthernet1/0/13
switchport trunk allowed vlan 1-3
channel-group 6 mode auto
!
interface GigabitEthernet1/0/14
switchport trunk allowed vlan 1-3
channel-group 6 mode auto
!
interface GigabitEthernet1/0/15
!
interface GigabitEthernet1/0/16
!
interface GigabitEthernet1/0/17
!
interface GigabitEthernet1/0/18
!
interface GigabitEthernet1/0/19
!
interface GigabitEthernet1/0/20
!
interface GigabitEthernet1/0/21
!
interface GigabitEthernet1/0/22
!
interface GigabitEthernet1/0/23
!
interface GigabitEthernet1/0/24
!
interface Vlan1
ip address 192.168.0.1 255.255.255.0
!
interface Vlan2
no ip address
!
interface Vlan3
no ip address
!
ip classless
ip http server
!
!
control-plane
!
!
line con 0
line vty 0 4
password netezza
login
line vty 5 15
password netezza
login
!
!
end
 
Old 12-07-2008, 04:45 PM   #2
acid_kewpie
Moderator
 
Registered: Jun 2001
Location: UK
Distribution: Gentoo, RHEL, Fedora, Centos
Posts: 43,398

Rep: Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963
not really sure what your issue is here. you can do an RR for the outbound data, but once it's in the switch, it's using the 3750's mac based algorithm by default which uses only only one link per mac. note that it's mac by default there, and you can (afair) change it to ip address instead, pulling it up from l2 to l3.
 
Old 12-08-2008, 01:06 PM   #3
zerobane
Member
 
Registered: Jan 2006
Posts: 41

Original Poster
Rep: Reputation: 16
So is there anyway to get a 2gigabit connection from host a to host b through the switch?
 
Old 12-08-2008, 01:20 PM   #4
acid_kewpie
Moderator
 
Registered: Jun 2001
Location: UK
Distribution: Gentoo, RHEL, Fedora, Centos
Posts: 43,398

Rep: Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963
not on a 3750, no. Do you really have a need for this? I'd thought that any app with that much throughput should be clustered across multiple boxes anyway. I believe there are theoretical issues about out of sequence streams and such if you are blasting so much data across multiple links. As the packets would be transmitted over the links as fast as possible after buffering, they'd hit each destination nic separately and just about every packet would be out of order I'd think. and as you naturally would have to recover that in software not within a single nic like you would do, that'd really suck the big one.
 
Old 12-08-2008, 05:10 PM   #5
zerobane
Member
 
Registered: Jan 2006
Posts: 41

Original Poster
Rep: Reputation: 16
Well the end goal was to make a appnote for customers so I could accurately describe the different bond modes in linux. Mainly round robin / LACP and software bonding with ALB / TLB.

The app we run is actaully clustered across 100+ blades, but intial data loads go through a single host then up to through the 3750 to the 100+ blades...

As you suggested I changed the port-channel load-balance src-dst-ip seemed to fix the issue. Along with setting the channel groups = on rather then pagp. (thanks!)

results now look like:


round-robin 500 meg file

[root@nps26002 ~]# time cat test_500MB | nc 192.168.0.125 7778

real 0m2.040s
user 0m0.043s
sys 0m0.656s
[root@nps26002 ~]#



!
version 12.2
no service pad
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
!
hostname Switch
!
enable secret 5 $1$RTrZ$314Aa.ue.b8WxaKFg.psz0
!
no aaa new-model
switch 1 provision ws-c3750g-24t
vtp domain ciscovtp
vtp mode transparent
ip subnet-zero
!
!
!
!
port-channel load-balance src-dst-ip
no file verify auto
spanning-tree mode pvst
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
interface Port-channel1
description rr channel bundle
switchport trunk encapsulation dot1q
switchport mode trunk
spanning-tree portfast trunk
!
interface Port-channel2
description rr channel bundle
switchport trunk encapsulation dot1q
switchport mode trunk
spanning-tree portfast trunk
!
interface Port-channel3
no switchport
no ip address
!
interface Port-channel4
switchport trunk allowed vlan 1-3
!
interface Port-channel5
!
interface Port-channel6
switchport trunk encapsulation dot1q
switchport mode trunk
!
interface GigabitEthernet1/0/1
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 1 mode on
spanning-tree portfast trunk
!
interface GigabitEthernet1/0/2
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 1 mode on
spanning-tree portfast trunk
!
interface GigabitEthernet1/0/3
!
interface GigabitEthernet1/0/4
!
interface GigabitEthernet1/0/5
!
interface GigabitEthernet1/0/6
!
interface GigabitEthernet1/0/7
!
interface GigabitEthernet1/0/8
!
interface GigabitEthernet1/0/9
!
interface GigabitEthernet1/0/10
!
interface GigabitEthernet1/0/11
!
interface GigabitEthernet1/0/12
!
interface GigabitEthernet1/0/13
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 2 mode on
spanning-tree portfast trunk
!
interface GigabitEthernet1/0/14
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 2 mode on
spanning-tree portfast trunk
!
interface GigabitEthernet1/0/15
!
interface GigabitEthernet1/0/16
!
interface GigabitEthernet1/0/17
!
interface GigabitEthernet1/0/18
!
interface GigabitEthernet1/0/19
!
interface GigabitEthernet1/0/20
!
interface GigabitEthernet1/0/21
!
interface GigabitEthernet1/0/22
!
interface GigabitEthernet1/0/23
!
interface GigabitEthernet1/0/24
!
interface Vlan1
ip address 192.168.0.1 255.255.255.0
!
interface Vlan2
no ip address
!
interface Vlan3
no ip address
!
ip classless
ip http server
!
!
control-plane
!
!
line con 0
line vty 0 4
password netezza
login
line vty 5 15
password netezza
login
!
!
end
 
Old 12-09-2008, 04:14 PM   #6
zerobane
Member
 
Registered: Jan 2006
Posts: 41

Original Poster
Rep: Reputation: 16
So would a L4 switch be capable of a host 1 to host 2 bond with 2 gigabit bandwidth?
 
Old 12-09-2008, 04:24 PM   #7
acid_kewpie
Moderator
 
Registered: Jun 2001
Location: UK
Distribution: Gentoo, RHEL, Fedora, Centos
Posts: 43,398

Rep: Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963
It's just not something i've seen any IOS device do. It could be useful to allocate based on port numbers, which i guess is what you mean, but that would naturally then require multiple (probably threaded) sessions, not just one. I don't believe any implementation of EtherChannel goes up to L4 though.
 
Old 12-10-2008, 01:02 PM   #8
zerobane
Member
 
Registered: Jan 2006
Posts: 41

Original Poster
Rep: Reputation: 16
That makes sense, route based on src xor dest port would result in correct routing on the recieve side as long as I am running mutiple sessions.

But IOS wont break up a single connection to two separate streams even if they came into two separate ports?
 
Old 12-10-2008, 01:04 PM   #9
zerobane
Member
 
Registered: Jan 2006
Posts: 41

Original Poster
Rep: Reputation: 16
btw first time using IOS so my lingo may be a bit off.
 
Old 03-02-2009, 08:04 AM   #10
mtsm
LQ Newbie
 
Registered: Mar 2006
Posts: 12

Rep: Reputation: 0
zero

i have the same issue , configured LACP , with 2 nics at 1 gig each , configured bonding with mode=0 ...then enabled lacp at the switch with mode=4 and have the same results.....the speed was not increased by the channel

i tested two machines in the same network and both with the exactly the same configuration ....the results is the same as i use just one nic.
 
Old 03-04-2009, 01:15 PM   #11
zerobane
Member
 
Registered: Jan 2006
Posts: 41

Original Poster
Rep: Reputation: 16
I've noticed with LACP and mode=4 you have to hit two different destination IP's in order to get it to use mutiple network cards from the orignating host or use two different originating ip's. I just used aliased ip's to fake this out...

Round robin and mode=0 always stripes across both network cards going out to the router but the destination and if its uses 2 etherchannels is purely controlled by the router.
 
Old 03-05-2009, 07:39 AM   #12
mtsm
LQ Newbie
 
Registered: Mar 2006
Posts: 12

Rep: Reputation: 0
Quote:
Originally Posted by zerobane View Post
I've noticed with LACP and mode=4 you have to hit two different destination IP's in order to get it to use mutiple network cards from the orignating host or use two different originating ip's. I just used aliased ip's to fake this out...

Round robin and mode=0 always stripes across both network cards going out to the router but the destination and if its uses 2 etherchannels is purely controlled by the router.

Zero , did you manage to make the LACP with sum bandwith ? i mean make the speed increase with two 1gig utpīs ? if i change the load-balance algorith to src-dst-ip it will increase my bandwith?
 
Old 03-05-2009, 09:03 AM   #13
acid_kewpie
Moderator
 
Registered: Jun 2001
Location: UK
Distribution: Gentoo, RHEL, Fedora, Centos
Posts: 43,398

Rep: Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963Reputation: 1963
The issue is not so much the linux side, but the switch side. You might be able to push data out of your bonded machine at 2gig, but that same rate then has to leave your switch to the destination server, and so that's totally down to the switch as to how a bonded LACP link there is used.
 
Old 11-22-2009, 08:34 AM   #14
jheusala
LQ Newbie
 
Registered: Nov 2009
Location: Oulu, Finland
Distribution: Debian
Posts: 2

Rep: Reputation: 0
Maximizing throughput with bonding mode 0 and VLANs with single switch

Quote:
Originally Posted by acid_kewpie View Post
The issue is not so much the linux side, but the switch side. You might be able to push data out of your bonded machine at 2gig, but that same rate then has to leave your switch to the destination server, and so that's totally down to the switch as to how a bonded LACP link there is used.
The problem indeed is that switches like my HP Procurve 2510G are trying to do load balancing based on source IP, destination IP and MAC -- which ends up only using one link at the time with maximum throughput 1000 Mbps (actually something like 976 Mbps in practice).

Fortunately you can split one switch into two with VLANs and almost double the throughput. (Best solution would be using two or more switches instead of one.)

TCP test:

Code:
$ cat /dev/zero|iperf -f m -c fs2 -d -t 30 -I
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 0.08 MByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to fs2, TCP port 5001
TCP window size: 0.02 MByte (default)
------------------------------------------------------------
[  4] local 192.168.9.26 port 60515 connected with 192.168.9.12 port 5001
[  5] local 192.168.9.26 port 5001 connected with 192.168.9.12 port 39065
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-30.0 sec  4329 MBytes  1211 Mbits/sec
[  5]  0.0-30.0 sec  5838 MBytes  1632 Mbits/sec
UDP test:

Code:
$ cat /dev/zero|iperf -f m -c fs2 -d -t 30 -I -u -b 2000M
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 0.12 MByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to fs2, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 0.12 MByte (default)
------------------------------------------------------------
[  4] local 192.168.9.26 port 40913 connected with 192.168.9.12 port 5001
[  3] local 192.168.9.26 port 5001 connected with 192.168.9.12 port 55976
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-30.0 sec  4639 MBytes  1297 Mbits/sec
[  4] Sent 3308950 datagrams
[  4] Server Report:
[ ID] Interval       Transfer     Bandwidth       Jitter   Lost/Total Datagrams
[  4]  0.0-30.0 sec  4562 MBytes  1276 Mbits/sec  0.017 ms 54460/3308947 (1.6%)
[  4]  0.0-30.0 sec  840703 datagrams received out-of-order
[  3]  0.0-30.0 sec  6459 MBytes  1806 Mbits/sec  0.064 ms 269336/4876894 (5.5%)
[  3]  0.0-30.0 sec  3239882 datagrams received out-of-order
I was using Linux bonding mode0 and Procurve trunk mode "static trunk" (which is non-LACP mode like Linux mode0 is too).

Mode 0 has still the problem that some datagrams will be received out-of-order and I think that using only one switch will increase this effect more... I'll do more tests about that once I get a second switch sometime in the future.

I also tested TCP with 4 VLANs:

Code:
$ cat /dev/zero|iperf -f m -c fs2 -d -t 30 -I
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 0.08 MByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to fs2, TCP port 5001
TCP window size: 0.02 MByte (default)
------------------------------------------------------------
[  4] local 192.168.9.26 port 51992 connected with 192.168.9.12 port 5001
[  5] local 192.168.9.26 port 5001 connected with 192.168.9.12 port 47836
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-30.0 sec  3464 MBytes    968 Mbits/sec
[  5]  0.0-30.0 sec  7970 MBytes  2228 Mbits/sec
..and UDP with 4 VLANs:

Code:
$ cat /dev/zero|iperf -f m -c fs2 -d -t 30 -I -u -b 4000M
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 0.12 MByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to fs2, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 0.12 MByte (default)
------------------------------------------------------------
[  4] local 192.168.9.26 port 55435 connected with 192.168.9.12 port 5001
[  3] local 192.168.9.26 port 5001 connected with 192.168.9.12 port 40210
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-30.0 sec  4157 MBytes  1162 Mbits/sec
[  4] Sent 2965331 datagrams
[ ID] Interval       Transfer     Bandwidth       Jitter   Lost/Total Datagrams
[  3]  0.0-30.0 sec  5475 MBytes  1531 Mbits/sec  0.016 ms 3051878/6957175 (44%)
[  3]  0.0-30.0 sec  2296743 datagrams received out-of-order
[  4] Server Report:
[  4]  0.0-30.0 sec  3585 MBytes  1002 Mbits/sec  0.021 ms 408047/2965331 (14%)
[  4]  0.0-30.0 sec  563499 datagrams received out-of-order
It doesn't look like bonding with 4 VLANs would be much better than with 2 VLANs on single switch.

Last edited by jheusala; 11-22-2009 at 08:39 AM.
 
Old 12-01-2009, 03:17 PM   #15
jheusala
LQ Newbie
 
Registered: Nov 2009
Location: Oulu, Finland
Distribution: Debian
Posts: 2

Rep: Reputation: 0
Two switches

Quote:
Originally Posted by jheusala View Post
Best solution would be using two or more switches instead of one.
Here's some test results with two switches:

Code:
jhh@vm6:~$ nttcp -n131072 -l16384 -D -T fs2.san
     Bytes  Real s   CPU s Real-MBit/s  CPU-MBit/s   Calls  Real-C/s   CPU-C/s
l2147483648    9.18    9.18   1870.4880   1870.5134  131072  14270.69   14270.9
12147483648    9.18    6.68   1870.4792   2571.8367  160607  17486.28   24043.0

jhh@vm6:~$ nttcp -r -n131072 -l16384 -D -T fs2.san
     Bytes  Real s   CPU s Real-MBit/s  CPU-MBit/s   Calls  Real-C/s   CPU-C/s
l2147483648    8.13    7.94   2112.9366   2164.6670  133152  16376.24   16777.2
12147483648    8.13    6.75   2113.8385   2545.1658  131072  16127.31   19418.1

jhh@vm6:~$ nttcp -u -r -n131072 -l16384 -D -T fs2.san
     Bytes  Real s   CPU s Real-MBit/s  CPU-MBit/s   Calls  Real-C/s   CPU-C/s
l   425984    4.83    0.00      0.7062    851.9680      27      5.60    6750.0
12147483648    4.82    2.50   3561.0618   6871.9477  131075  27169.37   52430.0
I had to change the test program because I got "broken pipe" errors with iperf (might be a bug).

All tests I've made with four 1 Gbps links have been limited roughly to 2 Gbps with TCP/IP -- even when using two pairs of switches. I don't know reason for that.

I also made a diagram documenting this current VLAN-based setup: http://nor.fi/jheusala/tmp/20091124-Switch-Diagram2.png
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
dns round-robin tzkolinux Linux - Networking 1 12-27-2004 03:49 AM
Setting Up DNS Round Robin? Longhair Linux - Networking 0 05-26-2003 01:34 AM
Setting Up DNS Round Robin Longhair Linux - Newbie 0 05-25-2003 12:09 PM
round robin for servers deepagodkhindi Programming 5 05-11-2003 05:50 AM
Round Robin DNS Lucsi Linux - Software 1 07-18-2002 04:17 AM


All times are GMT -5. The time now is 03:08 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration