Concourse cannot reach Docker registry

Question

I've set up a Concourse server (on bare metal), and everything is working swimmingly, except that I can't start any jobs and at this point I'm at a complete loss. Any help or suggestions would be greatly appreciated!

Taking the tutorial's hello world as an example, I get the following error:

$ fly -t tutorial execute -c task_hello_world.yml
executing build 55 at https://{{full hostname}}/builds/55 
initializing
resource script '/opt/resource/check []' failed: exit status 1

stderr:
failed to ping registry: 2 error(s) occurred:

* ping https: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
* ping http: Get http://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

errored

Grepping through the logs for any mention of docker, I get the following (and no other errors in the syslog):

Jan 11 03:39:11 {{host}} concourse[13704]: {"timestamp":"1515641951.825016260","source":"worker","message":"worker.garden.extract-resources.extract.already-extracted","log_level":1,"data":{"resource-type":"docker-image","session":"2.1.6"}}

Jan 11 03:41:55 {{host}} concourse[13613]: {"timestamp":"1515642115.660775423","source":"atc","message":"atc.create-build.do.task.image.failed-to-get-latest-image-version","log_level":2,"data":{"container":"ce9a7c49-210d-4cf7-6786-d27b1046c76b","error":"resource script '/opt/resource/check []' failed: exit status 1\n\nstderr:\nfailed to ping registry: 2 error(s) occurred:\n\n* ping https: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n* ping http: Get http://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n","session":"240.3.1.2"}}

I've started my web target with:

/usr/share/concourse/bin/concourse \
  web \
  --basic-auth-username {{username}} \
  --basic-auth-password {{password}} \
  --session-signing-key /usr/share/concourse/keys/session_signing_key \
  --tsa-host-key /usr/share/concourse/keys/tsa_host_key \
  --tsa-authorized-keys /usr/share/concourse/keys/authorized_keys \
  --bind-port 8080 \
  --tls-bind-port 8443 \
  --tls-key /etc/letsencrypt/live/{{full hostname}}/privkey.pem \
  --tls-cert /etc/letsencrypt/live/{{full hostname}}/fullchain.pem \
  --postgres-data-source postgres://concourse:concourse@localhost/atc \
  --external-url https://{{full hostname}}

...and my worker with:

/usr/share/concourse/bin/concourse \
  worker \
  --garden-dns-server 8.8.8.8 \
  --work-dir /usr/share/concourse/workspace \
  --tsa-host {{full hostname}} \
  --tsa-public-key /usr/share/concourse/keys/tsa_host_key.pub \
  --tsa-worker-private-key /usr/share/concourse/keys/worker_key

Docker appears to work fine (e.g., docker run hello-world), as does logging in with docker login. The worker nodes seem happy, and even register that they know about some containers (!?):

$ fly -t tutorial workers
name    containers platform  tags  team  state    version 
worker  3           linux    none  none  running  1.2

I have iptables routing ports 80 and 443 to 8080 and 8443, but there doesn't appear to be any difference if I turn this off and use the secondary ports. Curl seems to work fine, which I take to suggest that it's not a DNS issue:

$ curl https://registry-1.docker.io/v2/
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

Again, any suggestions are appreciated!

Answer 1

I have been facing this issue for a while and tried almost everything like disabling firewall,changing iptables settings.

Finally while playing around with docker-compose.yml file to set up concourse, I changed the following 2 settings in the file and resolved the issue.

CONCOURSE_GARDEN_DNS_PROXY_ENABLE=true
CONCOURSE_WORKER_GARDEN_DNS_PROXY_ENABLE=true

Hope this resolves your issue as well.