I'm having an issue where a server in CA (1000/full) and in VA (100/full) have very lopsided data transfer.

CA -> VA with iperf shows ~20Mbps
VA -> CA with iperf shows ~93Mbps

If we change the CA server to 100/FULL, transfer speed is 93Mbps both ways.

Some tuning was done to TCP window scaling parameters, but it won't correct the issue, just improve the CA -> VA numbers to what is listed above. I will say, turning TCP window scaling OFF will lower the transfer speed both ways to < 20Mbps.

The only clue I have when looking at wireshark dumps is that the window scale going OUT would never go past 10240 (scale is 8, so 2^8 x 40bytes). In the opposite direction, the window size will go above 3MB (scaled).

It is not a bandwidth problem as iperf with UDP shows 93Mbps both ways. Local transfers (CA 1000/full to CA 100/full) show full speed both ways, so I feel it is strictly related to TCP window scaling.

RedHat 5 64-bit on both sides. Any ideas why it won't scale above 10240?

Do you have firewalls in between ? .. also are you seeing drops/retransmits when CA == 1000/full ?

There was a firewall, but we tested without it as well. The logs are here if you want to look at them via wireshark :

http://209.207.216.188/pcap/

(I took a tshark dump on both sides - "fast" is when I pull a file in to CA and "slow" is when I push a file out to VA)

Look at the slow-CA.pcap and you'll see the window size never goes above 10240.

Could you reset the tcp tuning that was done then try using these settings :-

net.core.rmem_default = 262144
net.core.wmem_default = 262144
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

I was also wondering if any of the offload settings had been modified on the nic's .. ?

My previous values :

net.core.rmem_default=129024
net.core.wmem_default=129024
net.core.rmem_max=131071
net.core.wmem_max=131071
net.ipv4.tcp_rmem="4096 87380 4194304"
net.ipv4.tcp_wmem="4096 16384 4194304"

I set the values you requested. BEFORE :

-----
SJC_host# scp large root@VA_host:/usr/local/src/large2
large 100% 100MB 1.5MB/s 01:08
-----

AFTER:

-----
SJC_host# !scp
scp large root@VA_host:/usr/local/src/large2
large 100% 100MB 1.7MB/s 00:59
-----

The offload engine should be on :

----
# ethtool -k eth2 (interface the bond is running off of - yes, we tested w/o bonding)
Offload parameters for eth2:
Cannot get device udp large send offload settings: Operation not supported
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: off
generic-receive-offload: off
----

I played with turning off SACK as well - performance was terrible with it off.

How do the tuning values compare on the VA side ?

Apologies to all. I was about 100% certain we had tested outside of the firewall. The NOC is telling me this is an issue with the Nokia CheckPoint firewall which has been escalated to the vendor.

.. no probs