LinuxQuestions.org - 10Gb networking issue

Hi,

I have an issue with a CentOS 6.7 physical implementation on a HP DL360 G9, while running iperf from this server to another server the bandwidth fluctuate instead of producing a stable stream.

You can see below the bandwidth varies wildly instead of sending steadily

We have tried unbonding the NICS (they are in a LACP bond now), but it didn't make any difference, we have also tried to upgrade to the latest Intel ixgbe drivers but it didn't make nay difference either

The NIC ports are plugged into 2 Cisco Nexus 5672up

NICS:
2 x 10Gb port Intel Nic bonded in LACP.
driver: ixgbe
version: 4.0.1-k
firmware-version: 0x800006fd
bus-info: 0000:05:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

Kernel version
2.6.32-573.26.1.el6.x86_64

See below for iperf results

Code:

 [root@localhost ~]# iperf -c server1 -P 5 -i 1

------------------------------------------------------------

Client connecting to server1, TCP port 5001

TCP window size:  171 KByte (default)

------------------------------------------------------------

[  3] local 10.202.10.51 port 44790 connected with 10.204.10.4 port 5001

[  5] local 10.202.10.51 port 44787 connected with 10.204.10.4 port 5001

[  6] local 10.202.10.51 port 44788 connected with 10.204.10.4 port 5001

[  7] local 10.202.10.51 port 44789 connected with 10.204.10.4 port 5001

[  4] local 10.202.10.51 port 44791 connected with 10.204.10.4 port 5001

[ ID] Interval      Transfer    Bandwidth

[  3]  0.0- 1.0 sec  32.5 MBytes  273 Mbits/sec

[  5]  0.0- 1.0 sec  5.25 MBytes  44.0 Mbits/sec

[  6]  0.0- 1.0 sec  256 KBytes  2.10 Mbits/sec

[  3]  1.0- 2.0 sec  81.9 MBytes  687 Mbits/sec

[  5]  1.0- 2.0 sec  512 KBytes  4.19 Mbits/sec

[  7]  0.0- 1.0 sec  256 KBytes  2.10 Mbits/sec

[  4]  0.0- 1.0 sec  256 KBytes  2.10 Mbits/sec

[SUM]  0.0- 1.0 sec  38.5 MBytes  323 Mbits/sec

[  7]  1.0- 2.0 sec  128 KBytes  1.05 Mbits/sec

[  4]  1.0- 2.0 sec  128 KBytes  1.05 Mbits/sec

[  4]  2.0- 3.0 sec  10.9 MBytes  91.2 Mbits/sec

[  6]  1.0- 2.0 sec  640 KBytes  5.24 Mbits/sec

[SUM]  1.0- 2.0 sec  83.2 MBytes  698 Mbits/sec

[  6]  2.0- 3.0 sec  0.00 Bytes  0.00 bits/sec

[  7]  2.0- 3.0 sec  1.12 MBytes  9.44 Mbits/sec

[  4]  3.0- 4.0 sec  67.4 MBytes  565 Mbits/sec

[  3]  2.0- 3.0 sec  56.2 MBytes  472 Mbits/sec

[  3]  3.0- 4.0 sec  0.00 Bytes  0.00 bits/sec

[  5]  2.0- 3.0 sec  896 KBytes  7.34 Mbits/sec

[SUM]  2.0- 3.0 sec  69.1 MBytes  580 Mbits/sec

[  5]  3.0- 4.0 sec  0.00 Bytes  0.00 bits/sec

[  6]  3.0- 4.0 sec  768 KBytes  6.29 Mbits/sec

[  7]  3.0- 4.0 sec  768 KBytes  6.29 Mbits/sec

[SUM]  3.0- 4.0 sec  68.9 MBytes  578 Mbits/sec

[  3]  4.0- 5.0 sec  10.5 MBytes  88.1 Mbits/sec

[  7]  4.0- 5.0 sec  1.12 MBytes  9.44 Mbits/sec

[  5]  4.0- 5.0 sec  768 KBytes  6.29 Mbits/sec

[  6]  4.0- 5.0 sec  640 KBytes  5.24 Mbits/sec

[  4]  4.0- 5.0 sec  37.5 MBytes  315 Mbits/sec

[SUM]  4.0- 5.0 sec  50.5 MBytes  424 Mbits/sec

[  5]  5.0- 6.0 sec  14.0 MBytes  117 Mbits/sec

[  7]  5.0- 6.0 sec  1.12 MBytes  9.44 Mbits/sec

[  3]  5.0- 6.0 sec  74.6 MBytes  626 Mbits/sec

[  6]  5.0- 6.0 sec  384 KBytes  3.15 Mbits/sec

[  4]  5.0- 6.0 sec  384 KBytes  3.15 Mbits/sec

[SUM]  5.0- 6.0 sec  90.5 MBytes  759 Mbits/sec

[  5]  6.0- 7.0 sec  82.2 MBytes  690 Mbits/sec

[  3]  6.0- 7.0 sec  640 KBytes  5.24 Mbits/sec

[  6]  6.0- 7.0 sec  640 KBytes  5.24 Mbits/sec

[  7]  6.0- 7.0 sec  640 KBytes  5.24 Mbits/sec

[  4]  6.0- 7.0 sec  1.62 MBytes  13.6 Mbits/sec

[SUM]  6.0- 7.0 sec  85.8 MBytes  719 Mbits/sec

[  3]  7.0- 8.0 sec  60.2 MBytes  505 Mbits/sec

[  7]  7.0- 8.0 sec  6.50 MBytes  54.5 Mbits/sec

[  4]  7.0- 8.0 sec  768 KBytes  6.29 Mbits/sec

[  6]  7.0- 8.0 sec  768 KBytes  6.29 Mbits/sec

[  4]  8.0- 9.0 sec  62.6 MBytes  525 Mbits/sec

[  5]  7.0- 8.0 sec  7.62 MBytes  64.0 Mbits/sec

[SUM]  7.0- 8.0 sec  75.9 MBytes  636 Mbits/sec

[  5]  8.0- 9.0 sec  0.00 Bytes  0.00 bits/sec

[  7]  8.0- 9.0 sec  640 KBytes  5.24 Mbits/sec

[  3]  8.0- 9.0 sec  6.38 MBytes  53.5 Mbits/sec

[  4]  9.0-10.0 sec  28.2 MBytes  237 Mbits/sec

[  4]  0.0-10.0 sec  210 MBytes  176 Mbits/sec

[  6]  8.0- 9.0 sec  640 KBytes  5.24 Mbits/sec

[SUM]  8.0- 9.0 sec  70.2 MBytes  589 Mbits/sec

[  6]  9.0-10.0 sec  0.00 Bytes  0.00 bits/sec

[  6]  0.0-10.0 sec  4.75 MBytes  3.97 Mbits/sec

[  7]  9.0-10.0 sec  640 KBytes  5.24 Mbits/sec

[  7]  0.0-10.9 sec  13.0 MBytes  10.0 Mbits/sec

[  5]  9.0-10.0 sec  24.0 MBytes  201 Mbits/sec

[  5]  0.0-11.0 sec  135 MBytes  104 Mbits/sec

[  3]  9.0-10.0 sec  8.25 MBytes  69.2 Mbits/sec

[SUM]  9.0-10.0 sec  61.1 MBytes  513 Mbits/sec

[  3] 10.0-11.0 sec  0.00 Bytes  0.00 bits/sec

[  3]  0.0-11.3 sec  331 MBytes  247 Mbits/sec

[SUM]  0.0-11.3 sec  694 MBytes  517 Mbits/sec

This is what I would except, this is from a CentOS implementation on a virtual machine running on vSphere. It has 2 x 10 Gb Nics unbonded

Code:

[ ID] Interval      Transfer    Bandwidth

[  6]  0.0- 1.0 sec  31.6 MBytes  265 Mbits/sec

[  3]  0.0- 1.0 sec  23.6 MBytes  198 Mbits/sec

[  5]  0.0- 1.0 sec  22.5 MBytes  189 Mbits/sec

[  4]  0.0- 1.0 sec  18.9 MBytes  158 Mbits/sec

[  7]  0.0- 1.0 sec  16.1 MBytes  135 Mbits/sec

[SUM]  0.0- 1.0 sec  113 MBytes  946 Mbits/sec

[  6]  1.0- 2.0 sec  27.4 MBytes  230 Mbits/sec

[  3]  1.0- 2.0 sec  24.5 MBytes  206 Mbits/sec

[  5]  1.0- 2.0 sec  22.1 MBytes  186 Mbits/sec

[  4]  1.0- 2.0 sec  24.2 MBytes  203 Mbits/sec

[  7]  1.0- 2.0 sec  13.1 MBytes  110 Mbits/sec

[SUM]  1.0- 2.0 sec  111 MBytes  934 Mbits/sec

[  6]  2.0- 3.0 sec  16.9 MBytes  142 Mbits/sec

[  3]  2.0- 3.0 sec  24.2 MBytes  203 Mbits/sec

[  5]  2.0- 3.0 sec  22.2 MBytes  187 Mbits/sec

[  7]  2.0- 3.0 sec  28.2 MBytes  237 Mbits/sec

[  4]  2.0- 3.0 sec  20.1 MBytes  169 Mbits/sec

[SUM]  2.0- 3.0 sec  112 MBytes  937 Mbits/sec

[  6]  3.0- 4.0 sec  34.4 MBytes  288 Mbits/sec

[  3]  3.0- 4.0 sec  20.9 MBytes  175 Mbits/sec

[  5]  3.0- 4.0 sec  18.0 MBytes  151 Mbits/sec

[  4]  3.0- 4.0 sec  18.2 MBytes  153 Mbits/sec

[  7]  3.0- 4.0 sec  19.2 MBytes  161 Mbits/sec

[SUM]  3.0- 4.0 sec  111 MBytes  929 Mbits/sec

[  6]  4.0- 5.0 sec  21.6 MBytes  181 Mbits/sec

[  4]  4.0- 5.0 sec  22.8 MBytes  191 Mbits/sec

[  3]  4.0- 5.0 sec  21.5 MBytes  180 Mbits/sec

[  5]  4.0- 5.0 sec  21.5 MBytes  180 Mbits/sec

[  7]  4.0- 5.0 sec  24.4 MBytes  204 Mbits/sec

[SUM]  4.0- 5.0 sec  112 MBytes  937 Mbits/sec

[  4]  5.0- 6.0 sec  21.0 MBytes  176 Mbits/sec

[  3]  5.0- 6.0 sec  23.4 MBytes  196 Mbits/sec

[  5]  5.0- 6.0 sec  21.8 MBytes  182 Mbits/sec

[  6]  5.0- 6.0 sec  22.5 MBytes  189 Mbits/sec

[  7]  5.0- 6.0 sec  22.8 MBytes  191 Mbits/sec

[SUM]  5.0- 6.0 sec  111 MBytes  934 Mbits/sec

[  4]  0.0- 7.0 sec  145 MBytes  174 Mbits/sec

[  3]  6.0- 7.0 sec  24.1 MBytes  202 Mbits/sec

[  7]  6.0- 7.0 sec  24.2 MBytes  203 Mbits/sec

[  6]  6.0- 7.0 sec  20.8 MBytes  174 Mbits/sec

[  5]  6.0- 7.0 sec  22.6 MBytes  190 Mbits/sec

[  6]  7.0- 8.0 sec  28.8 MBytes  241 Mbits/sec

[  3]  7.0- 8.0 sec  24.1 MBytes  202 Mbits/sec

[  7]  7.0- 8.0 sec  27.5 MBytes  231 Mbits/sec

[  5]  7.0- 8.0 sec  30.1 MBytes  253 Mbits/sec

[  3]  8.0- 9.0 sec  27.4 MBytes  230 Mbits/sec

[  5]  8.0- 9.0 sec  33.2 MBytes  279 Mbits/sec

[  7]  8.0- 9.0 sec  24.1 MBytes  202 Mbits/sec

[  6]  8.0- 9.0 sec  26.2 MBytes  220 Mbits/sec

[  6]  9.0-10.0 sec  30.0 MBytes  252 Mbits/sec

[  6]  0.0-10.0 sec  260 MBytes  218 Mbits/sec

[  3]  9.0-10.0 sec  26.1 MBytes  219 Mbits/sec

[  3]  0.0-10.0 sec  240 MBytes  201 Mbits/sec

[  7]  9.0-10.0 sec  28.5 MBytes  239 Mbits/sec

[  7]  0.0-10.0 sec  228 MBytes  191 Mbits/sec

[  5]  9.0-10.0 sec  26.2 MBytes  220 Mbits/sec

[  5]  0.0-10.0 sec  240 MBytes  202 Mbits/sec

[SUM]  0.0-10.0 sec  1.09 GBytes  934 Mbits/sec

Any ideas?

Thanks

Regards