Cache Layers in GitLab Runner Docker Executor - Long Time DinD Container

Question

I'm working on GitLab CI at my project, and I created an image to make my tests and builds. When I ran it in docker executor every job needs to download images from the beginning. I need to cache layers and pulled images to improve my build and deploy time (5 min, up to 1 min with the insecure option).

I searched multiples links, and multiples articles, with many people with the same problem. But, the GitLab team does not resolve the problem. And the community doesn't have a solid and secure solution. The links below follow with the same problem:

1. The best answer don't work: Store layers in gitlab ci docker executor
2. Multiple changes to bypass the problem, but nothing work: https://blog.scottlogic.com/2018/02/09/multi-dind-ci-boxes.html
3. Discussion for don't use mounted docker.sock: https://gitlab.com/gitlab-org/gitlab-foss/issues/17769
4. Discussion for use mounted docker.sock: https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/
5. Build a container of long time (don't work with me): https://medium.com/@tonywooster/docker-in-docker-in-gitlab-runners-220caeb708ca
6. Documentation for don't mounted docker.sock: https://docs.gitlab.com/ce/ci/docker/using_docker_build.html#use-docker-in-docker-executor
7. Volumes config examples: https://github.com/ayufan/gitlab-ci-multi-runner/blob/master/docs/configuration/advanced-configuration.md#the-runnersdocker-section

The most likely approach (with layer caching) is to use a separate container and make the runner connect to it, and trigger executions from it. That way, all the layers would be in an "infinite life" container and not lose all cache at the end of a stage. The approach that considers exposing docker.sock as a mount is not only insecure but also has many problems with sharing files between containers, since they are all siblings, not parents and children who share volumes.

The approach to using an infinite life container would look something like this:

docker run --privileged --name gitlab-dind -d --restart=always  docker:19-dind --storage-driver=overlay2

or

docker network create gitlab-runner-net

docker run --privileged --name gitlab-runner-dind --network gitlab-runner-net --publish=2375:2375 --publish=2376:2376 -d docker:19-dind --storage-driver=overlay2

Then modify config.toml as follows:

[runners.docker]
  tls_verify = false
  image = "docker:19"   <--------
  privileged = false     <--------
  disable_cache = false
  volumes = ["/cache"]
  links = ["gitlab-runner-dind:docker"]   <-----------
  shm_size = 0
[runners.cache]

or respectively

[runners.docker]
  host = "tcp://gitlab-runner-dind:2375"    <--------
  tls_verify = false
  image = "docker:19"   <--------
  privileged = true     <--------
  disable_cache = false
  volumes = ["/cache"]
  network_mode = "gitlab-runner-net"   <-----------
  shm_size = 0
[runners.cache]

I'm tried with environment variable too (on config.toml and .gitlab-ci.yml):

DOCKER_TLS_CERTDIR=""
DOCKER_HOST=tcp://gitlab-runner-dind:2375

And remove from .gitlab-ci.yml:

services:
  - docker:19-dind
  alias: docker

The my current result is:

Running with gitlab-runner 12.4.1 (HASH)
  on NAME_OF_MY_RUNNER HASH
ERROR: Preparation failed: error during connect: Get http://gitlab-runner-dind:2375/v1.25/info: dial tcp: lookup gitlab-runner-dind on 172.31.0.2:53: no such host (executor_docker.go:980:0s)
Will be retried in 3s ...
ERROR: Preparation failed: error during connect: Get http://gitlab-runner-dind:2375/v1.25/info: dial tcp: lookup gitlab-runner-dind on 172.31.0.2:53: no such host (executor_docker.go:980:0s)
Will be retried in 3s ...
ERROR: Preparation failed: error during connect: Get http://gitlab-runner-dind:2375/v1.25/info: dial tcp: lookup gitlab-runner-dind on 172.31.0.2:53: no such host (executor_docker.go:980:0s)
Will be retried in 3s ...
ERROR: Job failed (system failure): error during connect: Get http://gitlab-runner-dind:2375/v1.25/info: dial tcp: lookup gitlab-runner-dind on 172.31.0.2:53: no such host (executor_docker.go:980:0s)

Using mounted docker.sock it works. But it is insecure, and the volumes have many problems to share files, artifacts, and cache.

root@GitlabRunner:/etc/gitlab-runner# gitlab-runner --version
Version:      12.4.1
Git revision: 05161b14
Git branch:   12-4-stable
GO version:   go1.10.8
Built:        2019-10-28T12:49:57+0000
OS/Arch:      linux/amd64

Answer 1

Maybe its better, when you use kaniko. Its not good, when you build with dind. (danger note @ gitlab reference)

kaniko offers the opportunity to use a cache-mechanism from the repository. kaniko@github

The only thing you need is a repository somewhere (I recommend Artifactory). With Artifactory you can also cache via the dind (see here).