Docker-Machine commands timeout TLS handshake but Dock Swarmer working ok

Question

I am having trouble with docker-machine now, it worked ok previously, as it timeouts for commands on machines running docker in Digital Ocean.

I can SSH to the machine fine and Docker Swarm and our system seems to be running ok on the machines (docker node command seem to work ok, e.g. docker node ls).

The problem seems just to be with docker machine. I had something similar in the past and it could be fixe by increasing the timeout but now that doesn't help.

The error seems to be caused by a net/http HTLS handshake time out as the output from:

docker-machine ls --timeout 30

shows:

NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
default - virtualbox Stopped Unknown 
dev-m0 - digitalocean Error Unknown Get https://api.digitalocean.com/v2/droplets/42100174: net/http: TLS handshake timeout
dev-w1 - digitalocean Error Unknown Get https://api.digitalocean.com/v2/droplets/42115817: net/http: TLS handshake timeout
... repeated for all managers and workers.

I am running Docker CE version 17.05.0-ce-rc1-mac8 Edge on my Mac and 17.03.0-ce on the Linux machines on Digital Ocean (it seems the latest available for them). I have swapped between Edge and Stable on the Mac to see if that was causing the problem but it didn't help.

I have restarted the machines (stop and start). I haven't regenerated certs because I can SSH into the machines with docker-machine ssh dev-m1 without a problem, so it didn't seem like certificates would be the problem to me. I am not trying to run any containers locally.

This has all been working fine in the past but just stopped recently.

Any help or suggestions most appreciated.

Thanks, Ashley.

Answer 1

Hypothesis

My guess is that the time inside your docker client VM is out of sync with the digital ocean server, and it is causing TLS handshake to fail. Try syncing the clock by running this command on your mac:

$ docker run --rm --privileged alpine hwclock -s

That command will set the clock inside the VM to the clock on your mac using the hwclock command. It needs privileged access because the container needs to read the time from the host hardware.

Explanation

The docker client on your mac is running inside a thin VM. The timing of the clock in the VM can get out of sync with the time on your mac and the time of the outside world, especially if you are using docker on a laptop that is allowed to sleep. That time de-sync can cause issues with any operations that need to know the time that events happened on the docker server (inside the VM) and compare that to events that happened outside the docker server. I suspect the TLS handshake with digital ocean is one such operation.

I experienced problems with docker's events not behaving the way I expected. After a long thread in the docker repository's issues section (moby/moby#25579) we figured out that the clock de-syncing was the cause.