How the gitlab-ci cache is working on docker runner? what is /cache directory? what is cache_dir?

Question

1. How the gitlab-ci cache is working on docker runner?

2. What is /cache directory?

3. What is cache_dir?

4. Where and how files matching the "paths" in "cache" gitlab-ci.yml are stored?

Answer 1

Volume mounted to /cache directory is created automatically on gitlab-runner installation and managed by cache_dir setting

more about cache_dir:

• https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section

• https://gitlab.com/gitlab-org/gitlab-runner/blob/main/docs/executors/docker.md#the-builds-and-cache-storage

If you modify the /cache storage path, you also need to make sure to mark this
directory as persistent by defining it in volumes = ["/my/cache/"] under the
[runners.docker] section in config.toml.

---

TLDR

/cache dir is different from cache config in gitlab-ci.yml

1. /cache dir in job container is where the cached files are stored
2. files matching to cache config in gitlab-ci.yml are copied to /cache/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME/<cache-key>-<cache-number> at the end of job
3. The "Clear Runner Caches" button in your project "Pipelines" page schedules TO NOT extract /cache/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME/<cache-key>-<cache-number>/cache.zip to dir specified in cache config in gitlab-ci.yml (Instead of "removing content of /cache folder as I though at first)

P.S.

There is container named gitlab-runner-cache created on machine with gitlab-runner (https://gitlab.com/gitlab-org/gitlab-runner/blob/af343971874198a1923352107409583b78e8aa80/executors/docker/executor_docker.go#L382)

(Seems like) This container is used to create anonymous volume where /cache data is stored. After the anonymous volume is created this container is stopped.

The job containers (meaning container were your tests typically run) mounts this anonymous volume

---

Proofs

HAVING gitlab-ci.yml

image: srghma/docker-nixos-with-git-crypt

cache:
  key: "test00000" # to reset cache - change this key OR clear cache in project settings page
  paths:
    - .mycache # gitlab allows only cache dirs that are relative to project root OR /cache (created automatically)

testtest:
  script:
    - nix-env -i tree

    - tree --dirsfirst -L 4 /cache
    - ls -al ./.mycache || true

    - echo "test" > /cache/test
    - mkdir -p ./.mycache
    - echo "test" > ./.mycache/test

    - tree --dirsfirst -L 4 /cache
    - ls -al ./.mycache || true

Output:

1. on first run

Running with gitlab-runner 11.6.0 (f100a208)
  on srghma_gitlab_runner 9b3980da
Using Docker executor with image srghma/docker-nixos-with-git-crypt ...
Pulling docker image srghma/docker-nixos-with-git-crypt ...
Using docker image sha256:ad3491aae178f629df713e0719750cc445b4881702b6b04b7cf325121f0032bf for srghma/docker-nixos-with-git-crypt ...
Running on runner-9b3980da-project-222-concurrent-0 via myrunner.com...
Fetching changes...
Removing .mycache/
HEAD is now at 675caa7 feat: cache update
From https://gitlab.com/srghma/myproject
   675caa7..3d1e223  nix        -> origin/nix
Checking out 3d1e2237 as nix...
Skipping Git submodules setup
Checking cache for test00000-11...
No URL provided, cache will be not downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache
$ nix-env -i tree
installing 'tree-1.8.0'
these paths will be fetched (0.03 MiB download, 0.09 MiB unpacked):
  /nix/store/dhfq0dsg9a0j5ai78bmh5qlrla8wvcxz-tree-1.8.0
copying path '/nix/store/dhfq0dsg9a0j5ai78bmh5qlrla8wvcxz-tree-1.8.0' from 'https://cache.nixos.org'...
building '/nix/store/dankqr2x4g5igc4w7lw9xqnn7lcy4f7a-user-environment.drv'...
created 233 symlinks in user environment
$ tree --dirsfirst -L 4 /cache
/cache

0 directories, 0 files
$ ls -al ./.mycache || true
$ echo "test" > /cache/test
ls: ./.mycache: No such file or directory
$ mkdir -p ./.mycache
$ echo "test" > ./.mycache/test
$ tree --dirsfirst -L 4 /cache
/cache
`-- test

0 directories, 1 file
$ ls -al ./.mycache || true
total 12
drwxr-xr-x    2 root     root          4096 Feb 24 11:44 .
drwxrwxrwx   20 root     root          4096 Feb 24 11:44 ..
-rw-r--r--    1 root     root             5 Feb 24 11:44 test
Creating cache test00000-11...
.mycache: found 2 matching files
No URL provided, cache will be not uploaded to shared cache server. Cache will be stored only locally.
Created cache
Job succeeded

2. on second run

Running with gitlab-runner 11.6.0 (f100a208)
  on srghma_gitlab_runner 9b3980da
Using Docker executor with image srghma/docker-nixos-with-git-crypt ...
Pulling docker image srghma/docker-nixos-with-git-crypt ...
Using docker image sha256:ad3491aae178f629df713e0719750cc445b4881702b6b04b7cf325121f0032bf for srghma/docker-nixos-with-git-crypt ...
Running on runner-9b3980da-project-222-concurrent-0 via myrunner.com...
Fetching changes...
Removing .mycache/
HEAD is now at 3d1e223 feat: cache update
Checking out 3d1e2237 as nix...
Skipping Git submodules setup
Checking cache for test00000-11...
No URL provided, cache will be not downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache
$ nix-env -i tree
installing 'tree-1.8.0'
these paths will be fetched (0.03 MiB download, 0.09 MiB unpacked):
  /nix/store/dhfq0dsg9a0j5ai78bmh5qlrla8wvcxz-tree-1.8.0
copying path '/nix/store/dhfq0dsg9a0j5ai78bmh5qlrla8wvcxz-tree-1.8.0' from 'https://cache.nixos.org'...
building '/nix/store/dankqr2x4g5igc4w7lw9xqnn7lcy4f7a-user-environment.drv'...
created 233 symlinks in user environment
$ tree --dirsfirst -L 4 /cache
/cache
|-- srghma
|   `-- myproject
|       `-- test00000-11
|           `-- cache.zip
`-- test

3 directories, 2 files
$ ls -al ./.mycache || true
total 12
drwxr-xr-x    2 root     root          4096 Feb 24 11:44 .
drwxrwxrwx   20 root     root          4096 Feb 24 11:44 ..
-rw-r--r--    1 root     root             5 Feb 24 11:44 test
$ echo "test" > /cache/test
$ mkdir -p ./.mycache
$ echo "test" > ./.mycache/test
$ tree --dirsfirst -L 4 /cache
/cache
|-- srghma
|   `-- myproject
|       `-- test00000-11
|           `-- cache.zip
`-- test

3 directories, 2 files
$ ls -al ./.mycache || true
total 12
drwxr-xr-x    2 root     root          4096 Feb 24 11:44 .
drwxrwxrwx   20 root     root          4096 Feb 24 11:44 ..
-rw-r--r--    1 root     root             5 Feb 24 11:44 test
Creating cache test00000-11...
.mycache: found 2 matching files
No URL provided, cache will be not uploaded to shared cache server. Cache will be stored only locally.
Created cache
Job succeeded

3. after you clear cache by clicking on "Clear Runner Caches" in your project "Pipelines" page

Running with gitlab-runner 11.6.0 (f100a208)
  on srghma_gitlab_runner 9b3980da
Using Docker executor with image srghma/docker-nixos-with-git-crypt ...
Pulling docker image srghma/docker-nixos-with-git-crypt ...
Using docker image sha256:ad3491aae178f629df713e0719750cc445b4881702b6b04b7cf325121f0032bf for srghma/docker-nixos-with-git-crypt ...
Running on runner-9b3980da-project-222-concurrent-0 via myrunner.com...
Fetching changes...
Removing .mycache/
HEAD is now at 3d1e223 feat: cache update
Checking out 3d1e2237 as nix...
Skipping Git submodules setup
Checking cache for test00000-12...
No URL provided, cache will be not downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache
$ nix-env -i tree
installing 'tree-1.8.0'
these paths will be fetched (0.03 MiB download, 0.09 MiB unpacked):
  /nix/store/dhfq0dsg9a0j5ai78bmh5qlrla8wvcxz-tree-1.8.0
copying path '/nix/store/dhfq0dsg9a0j5ai78bmh5qlrla8wvcxz-tree-1.8.0' from 'https://cache.nixos.org'...
building '/nix/store/dankqr2x4g5igc4w7lw9xqnn7lcy4f7a-user-environment.drv'...
created 233 symlinks in user environment
$ tree --dirsfirst -L 4 /cache
/cache
|-- srghma
|   `-- myproject
|       `-- test00000-11
|           `-- cache.zip
`-- test

3 directories, 2 files
$ ls -al ./.mycache || true
ls: ./.mycache: No such file or directory
$ echo "test" > /cache/test
$ mkdir -p ./.mycache
$ echo "test" > ./.mycache/test
$ tree --dirsfirst -L 4 /cache
/cache
|-- srghma
|   `-- myproject
|       `-- test00000-11
|           `-- cache.zip
`-- test

3 directories, 2 files
$ ls -al ./.mycache || true
total 12
drwxr-xr-x    2 root     root          4096 Feb 24 11:45 .
drwxrwxrwx   20 root     root          4096 Feb 24 11:45 ..
-rw-r--r--    1 root     root             5 Feb 24 11:45 test
Creating cache test00000-12...
.mycache: found 2 matching files
No URL provided, cache will be not uploaded to shared cache server. Cache will be stored only locally.
Created cache
Job succeeded