How to achieve gigabit speeds with Ubuntu using bonding?

Valetyn14 · 06-07-2013, 11:22 AM

Hi all..

I'm working with the driver bonding on Linux (Ubuntu 12.04.2 LTS).

I have 2 PC's with 2 NIC's (gigabit ethernet - Intel 82574L) in each one (in total, 4 NIC's). I want to improve my TCP throughput over a gigabit network, using bonding, but my problem is that I can't achieve gigabit speeds using bonding.

PC1 have eth1 and eth2 | PC2 have eth1 and eth2

My gigabit interface can reach 370 - 375 Mbps using a single gigabit interface (without using bonding). When I turn on my bonding, using mode 0 (balance-rr), for example, I can reach 760 - 772 Mbps (using iperf).

When I want to transfer data from one pc to another (using bonding mode balance-rr), I just can reach 472 Mbps (59 MB/s). To tranfer data, I use scp (A screenshot about my scp: http://imgur.com/qHf5quP).

My question is:

1 - What are the limitations of gigabit ethernet over Linux?

2 - Why can't I achieve gigabit speeds?

3 - Can anyone help me to change the default settings on Ubuntu for achieve gigabit speeds?

Can you advice me to tune my Ubuntu for achieve gigabit speeds?

My iperf screenshot (These values were obtained using bonding mode balance-rr (mode 0)):http://imgur.com/8IQWg9U

My sysctl.conf:

Code:

cat sysctl.conf

#

# /etc/sysctl.conf - Configuration file for setting system variables

# See /etc/sysctl.d/ for additional system variables

# See sysctl.conf (5) for information.

#

#kernel.domainname = example.com

# Uncomment the following to stop low-level messages on console

#kernel.printk = 3 4 1 3

##############################################################3

# Functions previously found in netbase

#

# Uncomment the next two lines to enable Spoof protection (reverse-path filter)

# Turn on Source Address Verification in all interfaces to

# prevent some spoofing attacks

#net.ipv4.conf.default.rp_filter=1

#net.ipv4.conf.all.rp_filter=1

# Uncomment the next line to enable TCP/IP SYN cookies

# See http://lwn.net/Articles/277146/

# Note: This may impact IPv6 TCP sessions too

#net.ipv4.tcp_syncookies=1

# Uncomment the next line to enable packet forwarding for IPv4

#net.ipv4.ip_forward=1

# Uncomment the next line to enable packet forwarding for IPv6

#  Enabling this option disables Stateless Address Autoconfiguration

#  based on Router Advertisements for this host

#net.ipv6.conf.all.forwarding=1


###################################################################

# Additional settings - these settings can improve the network

# security of the host and prevent against some network attacks

# including spoofing attacks and man in the middle attacks through

# redirection. Some network environments, however, require that these

# settings are disabled so review and enable them as needed.

#

# Do not accept ICMP redirects (prevent MITM attacks)

#net.ipv4.conf.all.accept_redirects = 0

#net.ipv6.conf.all.accept_redirects = 0

# _or_

# Accept ICMP redirects only for gateways listed in our default

# gateway list (enabled by default)

# net.ipv4.conf.all.secure_redirects = 1

#

# Do not send ICMP redirects (we are not a router)

#net.ipv4.conf.all.send_redirects = 0

#

# Do not accept IP source route packets (we are not a router)

#net.ipv4.conf.all.accept_source_route = 0

#net.ipv6.conf.all.accept_source_route = 0

#

# Log Martian Packets

#net.ipv4.conf.all.log_martians = 1

#

# IPv6

net.ipv6.conf.all.disable_ipv6 = 1

net.ipv6.conf.default.disable_ipv6 = 1

net.ipv6.conf.lo.disable_ipv6 = 1

Code:

ethtool -k eth1

Offload parameters for eth1:

rx-checksumming: on

tx-checksumming: on

scatter-gather: on

tcp-segmentation-offload: on

udp-fragmentation-offload: off

generic-segmentation-offload: on

generic-receive-offload: on

large-receive-offload: off

rx-vlan-offload: on

tx-vlan-offload: on

ntuple-filters: off

receive-hashing: off

Code:

ethtool -k eth2

Offload parameters for eth2:

rx-checksumming: on

tx-checksumming: on

scatter-gather: on

tcp-segmentation-offload: on

udp-fragmentation-offload: off

generic-segmentation-offload: on

generic-receive-offload: on

large-receive-offload: off

rx-vlan-offload: on

tx-vlan-offload: on

ntuple-filters: off

receive-hashing: off

Code:

ethtool -g eth1

Ring parameters for eth1:

Pre-set maximums:

RX:        4096

RX Mini:    0

RX Jumbo:    0

TX:        4096

Current hardware settings:

RX:        256

RX Mini:    0

RX Jumbo:    0

TX:        256

Code:

ethtool -g eth2

Ring parameters for eth2:

Pre-set maximums:

RX:        4096

RX Mini:    0

RX Jumbo:    0

TX:        4096

Current hardware settings:

RX:        256

RX Mini:    0

RX Jumbo:    0

TX:        256

suicidaleggroll · 06-08-2013, 08:35 PM

Have you looked at "top" while running your scp? scp transfer is encrypted, which requires a significant amount of overhead. Unless you're dealing with top of the line equipment, it's entirely possible the encryption overhead is what's holding your transfer speeds back. A quick look at top during the transfer would say for sure. 100% CPU usage means that's the problem. Upgrading your equipment or switching to a less demanding cypher (blowfish?) might help.

I have several servers under my care, from 8 year old dual single core machines to 2 year old dual hex core machines. On the exact same network, the older machines max out around 30 MB/s with scp, while the newer machines regularly hit ~70 MB/s, just due to processing power and encryption.

Beyond that, you may need to look at your networking equipment. Off the shelf "cheap" gigabit switches and routers will rarely let you actually hit gigabit speeds, but 60-70% of gigabit is relatively common.

Valetyn14 · 06-08-2013, 09:49 PM

Quote:

Originally Posted by suicidaleggroll

Have you looked at "top" while running your scp? scp transfer is encrypted, which requires a significant amount of overhead. Unless you're dealing with top of the line equipment, it's entirely possible the encryption overhead is what's holding your transfer speeds back. A quick look at top during the transfer would say for sure. 100% CPU usage means that's the problem. Upgrading your equipment or switching to a less demanding cypher (blowfish?) might help.

I have several servers under my care, from 8 year old dual single core machines to 2 year old dual hex core machines. On the exact same network, the older machines max out around 30 MB/s with scp, while the newer machines regularly hit ~70 MB/s, just due to processing power and encryption.

Beyond that, you may need to look at your networking equipment. Off the shelf "cheap" gigabit switches and routers will rarely let you actually hit gigabit speeds, but 60-70% of gigabit is relatively common.

Thank you very much for your answer.

I know that scp works with ssh ... but I thought that its encryptation was not as large.

I will take a look at 'top' while I'm transfering data to another pc. What do you recommend? Use top, without any flag?

Some clue?

jefro · 06-09-2013, 12:28 PM

Unless you have some ramdisk with this source data, you won't be able to get much more is my guess. The entire process is part of the transfer. From the slow hard drive to every other part in-between reduces total speeds.

Things I'd look at would be to remove the network switch and try it with a straight cable.

I'd set checksum offloading to the card.

As above, encryption causes some overhead.

Get rid of any and all other network services.

Some may play with the frames. Jumbo and super jumbo in some tests and situations increase speeds.