Transmit rate decreases when using both Intel NIC 10G ports

hjazz6 · 06-04-2012, 02:35 AM

Hi all,

I'm using the Intel 82599EB 10G NIC on a server running RHEL6. I ran a program where packets are generated at high rates. The NIC was previously installed on another PC where I was able to generate traffic at 8-9Gbps on each port concurrently. On this server, I was able to generate at 8-9Gbps only if one port is in use. If I were to generate on both ports concurrently, then my total transmit rate drops to about 12Gbps (~6Gbps on each port).

My relevant "lspci -vvv" output shows the following:

Code:

0b:00.0 Ethernet controller: ...
      Capabilities: [a0] Express (v2) Endpoint, MSI 00
            LnkCap: Port #1, Speed 5GT/s, Width x8
            LnkSta: Speed 5GT/s, Width x4
            LnkCtl2: Target Link Speed: 5GT/s

0b:00.1 Ethernet controller: ...
      Capabilities: [a0] Express (v2) Endpoint, MSI 00
            LnkCap: Port #1, Speed 5GT/s, Width x8
            LnkSta: Speed 5GT/s, Width x4
            LnkCtl2: Target Link Speed: 2.5GT/s

Is my transmit rate limited by the "x4" in LnkSta for both ports? And what does it mean to have "Target Link Speed: 5GT/s" on one port and "Target Link Speed: 2.5GT/s" on the other port? Does this contribute to the rate decrease too?

Also, I see a couple of PCIE slots that are longer than my NIC's PCIE connectors. Can I install my NIC on these longer slots instead?

Thank you.

Regards,
Rayne

baldy3105 · 06-05-2012, 12:54 PM

A V2 PCIE-X lane 5GT/s should be able to shift 500MBps which is 2Gbps. An x4 slot should therefore be able to move 8Gbps.

Whats wierd is the lane that is targeted at 2.5GT/s suggesting it is running in V1 mode for some reason. If you are bonding and therefore load balancing across two channels running at different rates the throughput will tend towards the lower.

Couldn't say why its doing this but I'd say thats where your problem is.

Try moving slots and seeing if it does the same thing.