Docker push - net/http: TLS handshake timeout

Question

I've deployed a private docker image registry on an AWS EC2 Ubuntu 14.04 instance. The registry is secured using Let's Encrypt certificate.

Unfortunately, I'm getting net/http: TLS handshake timeout for docker push operations that take longer than 300s:

This is the output of the time'd command:

[luqo33@home-pc containers]$ time docker push <my-registry-domain:5000>/nginx                                                    
The push refers to a repository [<my-registry-domain:5000>/nginx]
dda5a806f0b0: Layer already exists
ec35cfccb7f7: Layer already exists
94c1a232bb3f: Layer already exists
6d6b9812c8ae: Layer already exists
695da0025de6: Retrying in 1 second
fe4c16cbf7a4: Pushing [================================================>  ]   119 MB/123 MB
net/http: TLS handshake timeout

real    5m0.847s
user    0m0.097s
sys     0m0.017s

Logs of the regsitry:2 container do not show any errors - other than the notification that there was an unexpected EOF while receiving data. I can also push images that take less than 5min to push without problems.

I'm suspecting that it's a system setting at blame as the timeout happens always once the operations goes beyond 300 seconds. There isn't any load balancer or other proxy. <my-registry-domain:5000> points directly at the server IP.

How can I further investigate and possible remedy this situation?

EDIT

The same happens when I push images to other server providers (DigitalOcean), AWS ECS registry or even Docker Hub! I find it hard to believe that the Docker client would have a built-in handshake timeout of 300s.

I'm thinking that perhaps I should start look for the solution at the network level - with my hardware (wi-fi router) or my ISP.

Anybody has a clue what is happening here?

Answer 1

I got the same issue, this issue is may be from your internet connection, I solved it by decrementing the concurrency uploads (downloads for get) to 1 in dockerd. By using these args:

--max-concurrent-downloads  (default: 3)    Set the max concurrent downloads for each pull

--max-concurrent-uploads    (default: 5)    Set the max concurrent uploads for each push

uploading 5 images in same time may result a timeout if you have a low bandwidth.

https://docs.docker.com/engine/reference/commandline/dockerd/

Answer 2

Just adding a separate answer for those who may be dealing with it on a managed build environment like Azure Devops based on https://github.com/actions/virtual-environments/issues/2152#issuecomment-736325518

Change the setting as follows

sudo sed -i 's/ }/, \"max-concurrent-uploads\": 1 }/' /etc/docker/daemon.json
sudo systemctl restart docker