I noticed something that I did not understand in a Wireshark Capture of a data transfer. I have a long fat pipe (20 Mbits/sec, 720 Millisecond Round Trip Time). If I understand correctly, my TCP Send Buffer Size should be 16 KB. The contents of the /proc/sys/net/ipv4/tcp_wmem file is "4096 16384 1978368". Windows Scaling is enabled for the TCP Connection. The maximum TCP window size is 16 MB.

I thought that if the TCP Send Buffer Size was 16 KB, that the TCP socket could only send up to 16 KB of data that was not acknowledge (Bytes in Flight). But when I review the Wireshark capture, most of the time the TCP connection has around 256 KB in Flight.

Can someone explain this behavior? If Windows Scaling is enabled, the socket creates a large send buffer than the default value in the tcp_wmem file?

Thank You

The send buffer accumulates multiple writes before issuing send, so is more like the mimimum amount that will be in flight. Otherwise every write would cause a send, which would be a lot of overhead on some streaming protocols. Many send buffers worth of data may be in flight (Window size) so that bandwidth is used efficiently on long-latency connections.