docker login in custom gitlab-runner: No such host (http://docker:2375/v1.39/auth: dial tcp: lookup docker on 172.31.0.2:53: no such host)

Question

I have a private repo on gitlab.com. I have already setup a CI/CD pipeline using gitlab shared runners. On code push, a docker image is built, login to gitlab private docker registry and push the image. It works fine with the shared runners. Now I'm trying to create custom runners for the pipeline.

I've created a droplet on DigitalOcean (Docker 18.09.2~3 on Ubuntu 18.04 - 8 GB / 4 CPUs / 160 GB SSD)

ssh into the droplet

create a runner (from https://docs.gitlab.com/runner/install/docker.html#docker-image-installation-and-configuration)

docker run -d --name gitlab-runner --restart always \
  -v /srv/gitlab-runner/config:/etc/gitlab-runner \
  -v /var/run/docker.sock:/var/run/docker.sock \
  gitlab/gitlab-runner:latest

and register it (from https://docs.gitlab.com/runner/register/index.html#one-line-registration-command)

docker run --rm -t -i -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register \
  --non-interactive \
  --executor "docker" \
  --docker-image alpine:3 \
  --url "https://gitlab.com/" \
  --registration-token "MY_TOKEN" \
  --description "docker-runner" \
  --tag-list "docker,digitalocean" \
  --run-untagged="true" \
  --locked="false"

on running the pipeline, it fails at this command

docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY

also if I use

docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN registry.gitlab.com 

with this error message

error during connect: Post http://docker:2375/v1.39/auth: dial tcp: lookup docker on 67.207.67.2:53: no such host

I also get the same error if I use my local PC.

I have tried editing /etc/hosts with no success.

It still works with the shared runners, but fails with my custom runners. What am I missing?

docker info

Containers: 4
Running: 1
Paused: 0
 Stopped: 3
Images: 4
Server Version: 18.09.2
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
runc version: 09c8266bf2fcf9519a651b04ae54c967b9ab86ec
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-45-generic
Operating System: Ubuntu 18.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.79GiB
Name: docker-s-4vcpu-8gb-sgp1-01
ID: XF6Y:25TY:YXNO:RDHT:PEGU:KGR7:GPEM:DDBE:MAQ5:5VIJ:4GHB:Q7TF
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

this is the /etc/hosts file on the shared runner that processes the pipeline successfully

127.0.0.1   localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
127.0.0.1   0hshit.hopto.org
127.0.0.1   daymndaymn.myftp.org
127.0.0.1   loba.webhop.me
172.17.0.3  docker 50d9e6c88489 runner-72989761-project-11958054-concurrent-0-docker-0
172.17.0.4  runner-72989761-project-11958054-concurrent-0 

Answer 1

In GitLab 13.3 you can do the following in config.toml file of the runner:

Add in [[runners]] section:

  pre_build_script = "export DOCKER_HOST=tcp://docker:2375"
  environment = ["DOCKER_DRIVER=overlay2", "DOCKER_TLS_CERTDIR="]

and in [runners.docker] section

add:

tls_cert_path = ""

and change privileged to true.

In the end you will have something like this:

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "some_name"
  url = "https://my.url.com"
  token = "a_token"
  executor = "docker"
  pre_build_script = "export DOCKER_HOST=tcp://docker:2375"
  environment = ["DOCKER_DRIVER=overlay2", "DOCKER_TLS_CERTDIR="]
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_cert_path = ""
    tls_verify = false
    image = "ruby:2.6"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

In .gitlab-ci.yml

You only have to do this to user docker:

services:
  - docker:dind

Answer 2

I had a similar issue. Apparently, the dind service doesn't work with the 12.1 version of the gitlab runner. A temporary workaround can be found here: https://forum.gitlab.com/t/docker-dind-stops-working-after-12-1-0-update/28664/4

The (temporary) solution is to add the following snippet (from the link) in the correct place:

services:
    - name: docker:dind
      entrypoint: ["env", "-u", "DOCKER_HOST"]
      command: ["dockerd-entrypoint.sh"]
  variables:
    DOCKER_HOST: tcp://docker:2375/
    DOCKER_DRIVER: overlay2
    # See https://github.com/docker-library/docker/pull/166
    DOCKER_TLS_CERTDIR: ""

Answer 3

Had the same issue.

error during connect: Post http://docker:2375/v1.40/auth: dial tcp: lookup docker on 8.8.8.8:53: no such host

The issue also appeared on jekyll runner. Fixed it by disabling the jekyll runner (as it was redundant) and altering the newly implemented gitlab-runner's config located at:

/etc/gitlab-runner/config.toml

on a Debian 10 machine.

The error is, I believe, related to the docker version as mentioned in the previous comment and documented in gitlab's issue tracker:

    image = "docker:stable"

instead of:

    image = "debian:stable"

(using the default gitlab runner config installed on the debian machine with gitlab provided package). This line is located in the [[runners]] section.

The whole gitlab toml file:

concurrent = 1
check_interval = 5
log_level = "debug"

[session_server]
  session_timeout = 1800

[[runners]]
  name = "ecollect-docker-01"
  url = "https://mygitlabinstance.example.com/"
  token = "mys3cr3tt0ken"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "docker:stable"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
    shm_size = 0

Hope I've helped!