I'm learning Docker, installing docker 1.12.6 on my Red Hat Enterprise Linux 7.4 machine (Linux 3.10.0-693) and going through the 'getting started' exercise.
It's been a bit of a hassle getting it to work through my company's web proxy. (I don't know what sort of web proxy we have.)
I had to set up HTTP_PROXY/HTTPS_PROXY environment variables for dockerd, so it would use the proxy; and then add the company's CA certificates to the system's pki trusted store (since the web proxy does 'man in the middle' interposing of certificates for HTTPS connections). But finally I was able to load docker images from the docker.io registry; downloads to my machine all working fine.
But my attempts to do a 'docker push' to docker.io fail. Even with a tiny 13KB docker image the upload hangs and eventually a 'connection reset by peer' error is reported.
Doing a tcpdump the basic problem seems pretty clear. After a bit of traffic, setting things up, my docker client machine sends four HTTP packets in a row to the web proxy ... and it only sends three ACK packets back. One of the four outoing packets is never acknowledged. After this both sides re-send their last ACKs for a few minutes before one of them sends a FIN.
This is the interesting tail end of the tcpdump output:
Code:
...
# client sends HTTP to proxy, it sends an ACK back, everything fine.
17:17:17.132874 IP dockerclient.company.com.33484 > webproxy.local.webcache: Flags [P.], seq 326:484, ack 3372, win 37648, options [nop,nop,TS val 175751308 ecr 3912953706], length 158: HTTP
17:17:17.133929 IP webproxy.local.webcache > dockerclient.company.com.33484: Flags [.], ack 484, win 4863, options [nop,nop,TS val 3912953725 ecr 175751308], length 0
# Now the proxy relays a HTTP reply from upstream in two packets, my client sends a single ACK for both.
17:17:17.138783 IP webproxy.local.webcache > dockerclient.company.com.33484: Flags [P.], seq 3372:3378, ack 484, win 4863, options [nop,nop,TS val 3912953729 ecr 175751308], length 6: HTTP
17:17:17.138801 IP webproxy.local.webcache > dockerclient.company.com.33484: Flags [P.], seq 3378:3423, ack 484, win 4863, options [nop,nop,TS val 3912953729 ecr 175751308], length 45: HTTP
17:17:17.138941 IP dockerclient.company.com.33484 > webproxy.local.webcache: Flags [.], ack 3423, win 37648, options [nop,nop,TS val 175751314 ecr 3912953729], length 0
# Now we have the problem. My client sends four packets in a row to the proxy, with sequence numbers 1692, 3140, 3605, 3639 ...
17:17:17.139139 IP dockerclient.company.com.33484 > webproxy.local.webcache: Flags [P.], seq 484:1692, ack 3423, win 37648, options [nop,nop,TS val 175751314 ecr 3912953729], length 1208: HTTP
17:17:17.139157 IP dockerclient.company.com.33484 > webproxy.local.webcache: Flags [.], seq 1692:3140, ack 3423, win 37648, options [nop,nop,TS val 175751314 ecr 3912953729], length 1448: HTTP
17:17:17.139160 IP dockerclient.company.com.33484 > webproxy.local.webcache: Flags [P.], seq 3140:3605, ack 3423, win 37648, options [nop,nop,TS val 175751314 ecr 3912953729], length 465: HTTP
17:17:17.139179 IP dockerclient.company.com.33484 > webproxy.local.webcache: Flags [P.], seq 3605:3639, ack 3423, win 37648, options [nop,nop,TS val 175751314 ecr 3912953729], length 34: HTTP
# ... and the proxy sends separate ACKs for just three of the four, missing 3140
17:17:17.140457 IP webproxy.local.webcache > dockerclient.company.com.33484: Flags [.], ack 1692, win 6071, options [nop,nop,TS val 3912953731 ecr 175751314], length 0
17:17:17.140505 IP webproxy.local.webcache > dockerclient.company.com.33484: Flags [.], ack 3605, win 7984, options [nop,nop,TS val 3912953731 ecr 175751314], length 0
17:17:17.140513 IP webproxy.local.webcache > dockerclient.company.com.33484: Flags [.], ack 3639, win 8018, options [nop,nop,TS val 3912953731 ecr 175751314], length 0
# Everything hangs for 30 seconds; client resends its last ACK, proxy retransmits its last ACK.
17:17:47.142582 IP dockerclient.company.com.33484 > webproxy.local.webcache: Flags [.], ack 3423, win 37648, options [nop,nop,TS val 175781318 ecr 3912953731], length 0
17:17:47.143579 IP webproxy.local.webcache > dockerclient.company.com.33484: Flags [.], ack 3639, win 8018, options [nop,nop,TS val 3912983735 ecr 175751314], length 0
# Everything hangs for another 30 seconds; client resends its last ACK, so does proxy
17:18:17.152561 IP dockerclient.company.com.33484 > webproxy.local.webcache: Flags [.], ack 3423, win 37648, options [nop,nop,TS val 175811328 ecr 3912983735], length 0
17:18:17.153663 IP webproxy.local.webcache > dockerclient.company.com.33484: Flags [.], ack 3639, win 8018, options [nop,nop,TS val 3913013746 ecr 175751314], length 0
# ... and so forth
I'm stymied by this (but I am no networking expert). In the second set of packets above, starting from time 17:17:17.138783, the proxy sends two packets, my client then sends a single ACK packet back, acknowledging the greater/last of the sequence numbers, everything is fine. You'd think my client would be able to do the same thing when the proxy skips an explicit ACK of sequence number 3140 but sends two more for the *greater* sequence numbers of the last two packets of the group of four. But as you can see things halt, nothing happens until my client resends its last ACK every 30 seconds.
I'd appreciate any and all advice regarding the following questions:
Q1. Can anyone give me advice on which of the two machines is doing things 'wrong'? Out of curiosity I'm intrigued as to what the TCP/IP standard requires and which one is slipping up.
Q2. Are there configuration settings I can tweak to get Linux to avoid sending the four packets at once? Slow things down, serialise transmission, avoid sending queued packets until the previous transmission is acknowledged by the proxy? Thus stepping around this 'missing ACK packet' issue which I assume is the heart of my problem?
Q3. Is there any other way I can fix this and get docker working behind my corporate firewall? Anyone seen anything like this before, is it a common problem?
I can't affect/change the firewall in any way but I'm in control of the Linux docker client machine.
Many thanks for any advice!
