I've been struggling with this for a few days. My system has two identical hardware linux machines (except one is master and other is slave with diskless remote booting):

- Intel i5 3470 ivy
- 4GB Corsair RAM
- Gigabyte GA-B75M-HD3 microboard
- onboard Realtek Gigabit ethernet
- Trendnet 8-port Gigabit switch
- Cat5e cables

System should be capable of close to Gbps speeds. iperf (UDP mode) indicates I get ~0.9Gbps. But the iperf (TCP mode) has issues. When I first bootup both machines and run iperf tests I get ~0.9Gbps but the second time I run the exact same iperf I only get 70 Mbps.

I've been reading up on TCP protocol and am somewhat familiar with congestion control, window sizes, MSS, latency, etc. One observation that I made was the following:

command speed
-------------------------------------------
[1] iperf -c davinci103 -w 11k 70 Mbps
[2] iperf -c davinci103 -w 22k 140 Mbps

wireshark capture is below for command [2]:

No. Time Source Destination Protocol Length Info
18 0.000778 192.168.2.3 192.168.2.1 TCP 66 5001 > 48161 [ACK] Seq=1 Ack=8713 Win=31872 Len=0 TSval=3157080 TSecr=3249288
19 0.000804 192.168.2.1 192.168.2.3 TCP 1514 48161 > 5001 [ACK] Seq=14505 Ack=1 Win=14720 Len=1448 TSval=3249288 TSecr=3157080
20 0.000807 192.168.2.1 192.168.2.3 TCP 1514 48161 > 5001 [PSH, ACK] Seq=15953 Ack=1 Win=14720 Len=1448 TSval=3249288 TSecr=3157080
21 0.000810 192.168.2.3 192.168.2.1 TCP 66 5001 > 48161 [ACK] Seq=1 Ack=14505 Win=42240 Len=0 TSval=3157080 TSecr=3249288
22 0.000826 192.168.2.1 192.168.2.3 TCP 1514 48161 > 5001 [ACK] Seq=17401 Ack=1 Win=14720 Len=1448 TSval=3249288 TSecr=3157080
23 0.001028 192.168.2.1 192.168.2.3 TCP 1514 48161 > 5001 [ACK] Seq=18849 Ack=1 Win=14720 Len=1448 TSval=3249288 TSecr=3157080
24 0.001042 192.168.2.1 192.168.2.3 TCP 1514 48161 > 5001 [ACK] Seq=20297 Ack=1 Win=14720 Len=1448 TSval=3249288 TSecr=3157080
25 0.001237 192.168.2.1 192.168.2.3 TCP 1514 48161 > 5001 [ACK] Seq=21745 Ack=1 Win=14720 Len=1448 TSval=3249288 TSecr=3157080
26 0.001245 192.168.2.1 192.168.2.3 TCP 1514 48161 > 5001 [ACK] Seq=23193 Ack=1 Win=14720 Len=1448 TSval=3249288 TSecr=3157080
27 0.001414 192.168.2.1 192.168.2.3 TCP 1514 48161 > 5001 [PSH, ACK] Seq=24641 Ack=1 Win=14720 Len=1448 TSval=3249288 TSecr=3157080
28 0.001421 192.168.2.1 192.168.2.3 TCP 1514 48161 > 5001 [ACK] Seq=26089 Ack=1 Win=14720 Len=1448 TSval=3249288 TSecr=3157080
29 0.001572 192.168.2.3 192.168.2.1 TCP 66 5001 > 48161 [ACK] Seq=1 Ack=17401 Win=42240 Len=0 TSval=3157080 TSecr=3249288
30 0.001590 192.168.2.3 192.168.2.1 TCP 66 5001 > 48161 [ACK] Seq=1 Ack=26089 Win=42240 Len=0 TSval=3157080 TSecr=3249288

The latency in my lan (using ping) is ~0.2ms and in the above packet sequence it almost seems as if two packets are being set almost instantaneously and then it waits 0.2ms before send the next two packets. Again, this is when iperf -w option is 22k. If -w option is 11k the wireshark output looks very similar except every segment is followed by 0.2ms before sending next segment.

I read that the iperf -w isn't actually setting the window size. For all these tests the window size = 115 Bytes with window scaling = 7 (so 128x). Instead, the -w sets the buffer size? Not quite sure what this means yet.

Does anyone have insight into what might be my issue? Thank you so much!

Boot both systems to a cd where you can load to ram. Then connect them to each other with a direct cable. Be sure it is cat6 cable and tested. Then repeat some tests.

Also knowing more about the nic state on this would help.

By NIC state, do you mean ethtool results? Ethtool returns:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised pause frame use: Symmetric Receive-only
Advertised auto-negotiation: Yes
Link partner advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Link partner advertised pause frame use: Symmetric Receive-only
Link partner advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: MII
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Cannot get wake-on-lan settings: Operation not permitted
Current message level: 0x00000033 (51)
drv probe ifdown ifup
Link detected: yes

The output from :-
after you run the tests will also help

I've tried the ip -s link but can't find any noticeable differences before and after. Actually, I followed the suggestions on this website: http://www.cyberciti.biz/faq/linux-tcp-tuning/ and they seem to help. I changed my /etc/sysctl.conf file appending the following statements:

################################################################
# User added settings. For tuning TCP
net.core.wmem_max=12582912
net.core.rmem_max=12582912
net.ipv4.tcp_rmem= 10240 87380 12582912
net.ipv4.tcp_wmem= 10240 87380 12582912
net.ipv4.tcp_no_metrics_save = 1

Now I can consistently achieve 930Mbit/s. The problem is not fully resolved however bc if I run iperf -c davinci103 -d, then I get 930Mbit/s going from client -> server, but only 35Mbit/s going from server -> client. I am double-checking whether the sysctl.conf settings were also applied to the client machines.