[SOLVED] 10gbe ixgbe (intel X550T) slow at 420mbyte/s max, p2p lan cat8 1m, tried ethtool and ifconfig options to no avail

tinfoil3d · 04-10-2020, 03:01 AM

Hey LQ community, first time dealing with 10gbe.
I have two boxes, one is older with i7 2.8ghz from 2009 generation, pcie 2.0 and I just put the Intel X550T there to make use of absolutely similar but builtin adapter on my newer box with AMD EPYC gen 1.
I'm dling file from tmpfs on either box and never go above 415mbyte/s which is about half of potential bandwidth there, right?
I tried most stuff from https://www.kernel.org/doc/ols/2009/...es-169-184.pdf with a little improvement from 393mbyte/s to current 415.
that's the result of me setting mtu 9000, txqueuelen 10000.
Also tried this to no improvements over current result. https://darksideclouds.wordpress.com...ghway-to-hell/
And a few other pages out there on google, most stating same options.
I don't see a 100% load burst on i7 box when I dl file, it's around 10% across the cores. So CPU shouldn't be a bottleneck, cable(it's 1meter cat8, with a nice thick shielding, i tried flight-proven cat7 with similar speed, and tried both ports(there are 2 on each box)) shouldn't be a bottleneck and ram speed and pcie 2.0 speed shouldn't be.
iperf also reports same speed.
Tried these tests in both directions.
There's gotta be something obvious that I'm missing, right?

Pretty sure it doesn't matter but it's lfs on i7 and debian 10 on epyc, both sharing same sysctl settings and ethtool-controlled options.
lspci -n for it on both computers:

Code:

62:00.0 0200: 8086:1563 (rev 01)

so it's similar device. I bought this particular one for compatibility but didn't expect these issues.

Maybe someone here would suggest something, I'm out of ideas.
TIA

-- solved, see my own answers, I have way too many PCIe lanes used up on that i7 system. potential solution may be to use x16-x1 riser for GPU as I don't play games there so it will work and free up 4 lanes for NIC. But ultimately it's narrow margin with intel CPUs anyway. You can't plug too many devices and expect it would just work like it will on latest generation of amd processors.

dc.901 · 04-10-2020, 06:49 AM

There are several other factors to look at...
How is the IO load (on both side) when you are uploading/downloading?
What type of disk drives? And is there a RAID involved (software/hardware)?
How is the memory usage during up/download; is the it doing direct or cached IO?

tinfoil3d · 04-10-2020, 07:09 AM

it's from tmpfs, memory definitely is faster than 400mbyte/s, even ddr1. no swap.
iperf produces same results.
cpu load on i7 is minimal, on epyc it's not a load at all.

also, updating post with more details:
ethtool info for interfaces differs slightly but i'm pretty sure it doesn't matter: you can just see the diff.

this is for i7: https://paste.ubuntu.com/p/JmDQcYWkbs/
this is epyc: https://paste.ubuntu.com/p/ZThRF6kcqs/
and this is diff(embedded):

Code:

# diff i7 epyc 
1,2c1
< Cannot get device udp-fragmentation-offload settings: Operation not supported
< Features for eth101:
---
> Features for eth1:
8c7
< 	tx-checksum-fcoe-crc: off
---
> 	tx-checksum-fcoe-crc: on
32c31
< tx-fcoe-segmentation: off
---
> tx-fcoe-segmentation: on

and ethtool -g for both boxes match:

Code:

Ring parameters for eth1:
Pre-set maximums:
RX:		4096
RX Mini:	0
RX Jumbo:	0
TX:		4096
Current hardware settings:
RX:		4096
RX Mini:	0
RX Jumbo:	0
TX:		4096

tinfoil3d · 04-11-2020, 06:22 AM

i think i get it now: it's the PCIe available lane count that's messing with me. this i7 box has i7-860 cpu with 16 lanes max, running on mb with pcie 2.0. the box has 6 sata 2.0 drives connected, two legacy PCI slots used, one GPU (that is also throttled down from x16 to x8 speed according to dmesg), check this out:

Code:

[    4.559030] ixgbe 0000:02:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x1 link at 0000:00:05.0 (capable of 31.504 Gb/s with 8 GT/s x4 link)
[    5.611006] ixgbe 0000:02:00.1: 4.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x1 link at 0000:00:05.0 (capable of 31.504 Gb/s with 8 GT/s x4 link)

and this is how it should look like(this is from pcie 3.0 box with amd epyc)

Code:

[    2.364368] ixgbe 0000:62:00.0: 32.000 Gb/s available PCIe bandwidth (5 GT/s x8 link)

the rest of lanes go to 6 sata drives and pci devices and gpu, leaving us with max of x1 or maybe x2 lanes left which isn't gonna cut it. so the speeds i'm seeing are very likely to be quite good for this setup. you couldn't wish for more. if i could get rid of some peripherals then maybe i could have claimed x4 but that's also not possible, i need all those drives.
so, this pretty much brings the question to closure.
thanks goes to the-architect from https://www.unix.com/member_modal.ph...184144&modal=1 who gave me an idea to check all the dmesg.

tinfoil3d · 04-11-2020, 06:53 PM

there's also one more solution for those who have a lot in their boxes: you can use x16->x1 riser that miners use for GPU. that'll likely make it impossible to play games or hd videos but would work just fine for regular daily work and browsing. and that'll free up your lanes for devices that need em.
there may also be a way to explicitly tell your system which slot should be cut to what number of lanes through BIOS menu. it's not a common feature though.