How the gitlab-ci cache is working on docker runner? what is /cache directory? what is cache

How the gitlab-ci cache is working on docker runner?
What is /cache directory?
What is cache_dir?
Where and how files matching the "paths" in "cache" gitlab-ci.yml are stored?

Where is cache stored in GitLab runner?

By default, they are stored locally in the machine where the Runner is installed and depends on the type of the executor. Locally, stored under the gitlab-runner user's home directory: /home/gitlab-runner/cache/<user>/<project>/<cache-key>/cache. zip .

How does GitLab CI cache work?

all tiers. A cache is one or more files a job downloads and saves. Subsequent jobs that use the same cache don't have to download the files again, so they execute more quickly.

Where is GitLab runner build directory?

Build Directory GitLab Runner clones the repository to a path that exists under a base path better known as the Builds Directory. The default location of this base directory depends on the executor. For: Kubernetes, Docker and Docker Machine executors, it is /builds inside of the container.

How does the Docker GitLab Runner work?

GitLab Runner can use Docker to run jobs on user provided images. This is possible with the use of Docker executor. The Docker executor when used with GitLab CI, connects to Docker Engine and runs each build in a separate and isolated container using the predefined image that is set up in .

What is clear-Docker-cache in GitLab Runner?

GitLab Runner provides the clear-docker-cache script to remove old containers and volumes that can unnecessarily consume disk space. Run clear-docker-cache regularly (using cron once per week, for example), ensuring a balance is struck between: Maintaining some recent containers in the cache for performance. Reclaiming disk space.

Where is the job cache stored in GitLab?

All caches defined for a job are archived in a single cache.zip file. The runner configuration defines where the file is stored. By default, the cache is stored on the machine where GitLab Runner is installed. The location also depends on the type of executor.

What is cache in GitLab CI CD?

Caching in GitLab CI/CD. A cache is one or more files that a job downloads and saves. Subsequent jobs that use the same cache don’t have to download the files again, so they execute more quickly. To learn how to define the cache in your .gitlab-ci.yml file, see the cache reference .

Where is the cache in a docker container?

With Docker executor cache from jobs is stored in /cache dir in the container itself. Since the container is ephemeral this is not stored on Docker host out of the box. You need to configure your GitLab Runner for local cache.

Volume mounted to /cache directory is created automatically on gitlab-runner installation and managed by cache_dir setting

more about cache_dir:

https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section
https://gitlab.com/gitlab-org/gitlab-runner/blob/main/docs/executors/docker.md#the-builds-and-cache-storage

If you modify the /cache storage path, you also need to make sure to mark this
directory as persistent by defining it in volumes = ["/my/cache/"] under the
[runners.docker] section in config.toml.

TLDR

/cache dir is different from cache config in gitlab-ci.yml

/cache dir in job container is where the cached files are stored
files matching to cache config in gitlab-ci.yml are copied to /cache/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME/<cache-key>-<cache-number> at the end of job
The "Clear Runner Caches" button in your project "Pipelines" page schedules TO NOT extract /cache/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME/<cache-key>-<cache-number>/cache.zip to dir specified in cache config in gitlab-ci.yml (Instead of "removing content of /cache folder as I though at first)

P.S.

There is container named gitlab-runner-cache created on machine with gitlab-runner (https://gitlab.com/gitlab-org/gitlab-runner/blob/af343971874198a1923352107409583b78e8aa80/executors/docker/executor_docker.go#L382)

(Seems like) This container is used to create anonymous volume where /cache data is stored. After the anonymous volume is created this container is stopped.

The job containers (meaning container were your tests typically run) mounts this anonymous volume

Proofs

HAVING gitlab-ci.yml

image: srghma/docker-nixos-with-git-crypt

cache:
  key: "test00000" # to reset cache - change this key OR clear cache in project settings page
  paths:
    - .mycache # gitlab allows only cache dirs that are relative to project root OR /cache (created automatically)

testtest:
  script:
    - nix-env -i tree

    - tree --dirsfirst -L 4 /cache
    - ls -al ./.mycache || true

    - echo "test" > /cache/test
    - mkdir -p ./.mycache
    - echo "test" > ./.mycache/test

    - tree --dirsfirst -L 4 /cache
    - ls -al ./.mycache || true

Output:

on first run

Running with gitlab-runner 11.6.0 (f100a208)
  on srghma_gitlab_runner 9b3980da
Using Docker executor with image srghma/docker-nixos-with-git-crypt ...
Pulling docker image srghma/docker-nixos-with-git-crypt ...
Using docker image sha256:ad3491aae178f629df713e0719750cc445b4881702b6b04b7cf325121f0032bf for srghma/docker-nixos-with-git-crypt ...
Running on runner-9b3980da-project-222-concurrent-0 via myrunner.com...
Fetching changes...
Removing .mycache/
HEAD is now at 675caa7 feat: cache update
From https://gitlab.com/srghma/myproject
   675caa7..3d1e223  nix        -> origin/nix
Checking out 3d1e2237 as nix...
Skipping Git submodules setup
Checking cache for test00000-11...
No URL provided, cache will be not downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache
$ nix-env -i tree
installing 'tree-1.8.0'
these paths will be fetched (0.03 MiB download, 0.09 MiB unpacked):
  /nix/store/dhfq0dsg9a0j5ai78bmh5qlrla8wvcxz-tree-1.8.0
copying path '/nix/store/dhfq0dsg9a0j5ai78bmh5qlrla8wvcxz-tree-1.8.0' from 'https://cache.nixos.org'...
building '/nix/store/dankqr2x4g5igc4w7lw9xqnn7lcy4f7a-user-environment.drv'...
created 233 symlinks in user environment
$ tree --dirsfirst -L 4 /cache
/cache

0 directories, 0 files
$ ls -al ./.mycache || true
$ echo "test" > /cache/test
ls: ./.mycache: No such file or directory
$ mkdir -p ./.mycache
$ echo "test" > ./.mycache/test
$ tree --dirsfirst -L 4 /cache
/cache
`-- test

0 directories, 1 file
$ ls -al ./.mycache || true
total 12
drwxr-xr-x    2 root     root          4096 Feb 24 11:44 .
drwxrwxrwx   20 root     root          4096 Feb 24 11:44 ..
-rw-r--r--    1 root     root             5 Feb 24 11:44 test
Creating cache test00000-11...
.mycache: found 2 matching files
No URL provided, cache will be not uploaded to shared cache server. Cache will be stored only locally.
Created cache
Job succeeded

on second run

Running with gitlab-runner 11.6.0 (f100a208)
  on srghma_gitlab_runner 9b3980da
Using Docker executor with image srghma/docker-nixos-with-git-crypt ...
Pulling docker image srghma/docker-nixos-with-git-crypt ...
Using docker image sha256:ad3491aae178f629df713e0719750cc445b4881702b6b04b7cf325121f0032bf for srghma/docker-nixos-with-git-crypt ...
Running on runner-9b3980da-project-222-concurrent-0 via myrunner.com...
Fetching changes...
Removing .mycache/
HEAD is now at 3d1e223 feat: cache update
Checking out 3d1e2237 as nix...
Skipping Git submodules setup
Checking cache for test00000-11...
No URL provided, cache will be not downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache
$ nix-env -i tree
installing 'tree-1.8.0'
these paths will be fetched (0.03 MiB download, 0.09 MiB unpacked):
  /nix/store/dhfq0dsg9a0j5ai78bmh5qlrla8wvcxz-tree-1.8.0
copying path '/nix/store/dhfq0dsg9a0j5ai78bmh5qlrla8wvcxz-tree-1.8.0' from 'https://cache.nixos.org'...
building '/nix/store/dankqr2x4g5igc4w7lw9xqnn7lcy4f7a-user-environment.drv'...
created 233 symlinks in user environment
$ tree --dirsfirst -L 4 /cache
/cache
|-- srghma
|   `-- myproject
|       `-- test00000-11
|           `-- cache.zip
`-- test

3 directories, 2 files
$ ls -al ./.mycache || true
total 12
drwxr-xr-x    2 root     root          4096 Feb 24 11:44 .
drwxrwxrwx   20 root     root          4096 Feb 24 11:44 ..
-rw-r--r--    1 root     root             5 Feb 24 11:44 test
$ echo "test" > /cache/test
$ mkdir -p ./.mycache
$ echo "test" > ./.mycache/test
$ tree --dirsfirst -L 4 /cache
/cache
|-- srghma
|   `-- myproject
|       `-- test00000-11
|           `-- cache.zip
`-- test

3 directories, 2 files
$ ls -al ./.mycache || true
total 12
drwxr-xr-x    2 root     root          4096 Feb 24 11:44 .
drwxrwxrwx   20 root     root          4096 Feb 24 11:44 ..
-rw-r--r--    1 root     root             5 Feb 24 11:44 test
Creating cache test00000-11...
.mycache: found 2 matching files
No URL provided, cache will be not uploaded to shared cache server. Cache will be stored only locally.
Created cache
Job succeeded

after you clear cache by clicking on "Clear Runner Caches" in your project "Pipelines" page

Running with gitlab-runner 11.6.0 (f100a208)
  on srghma_gitlab_runner 9b3980da
Using Docker executor with image srghma/docker-nixos-with-git-crypt ...
Pulling docker image srghma/docker-nixos-with-git-crypt ...
Using docker image sha256:ad3491aae178f629df713e0719750cc445b4881702b6b04b7cf325121f0032bf for srghma/docker-nixos-with-git-crypt ...
Running on runner-9b3980da-project-222-concurrent-0 via myrunner.com...
Fetching changes...
Removing .mycache/
HEAD is now at 3d1e223 feat: cache update
Checking out 3d1e2237 as nix...
Skipping Git submodules setup
Checking cache for test00000-12...
No URL provided, cache will be not downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache
$ nix-env -i tree
installing 'tree-1.8.0'
these paths will be fetched (0.03 MiB download, 0.09 MiB unpacked):
  /nix/store/dhfq0dsg9a0j5ai78bmh5qlrla8wvcxz-tree-1.8.0
copying path '/nix/store/dhfq0dsg9a0j5ai78bmh5qlrla8wvcxz-tree-1.8.0' from 'https://cache.nixos.org'...
building '/nix/store/dankqr2x4g5igc4w7lw9xqnn7lcy4f7a-user-environment.drv'...
created 233 symlinks in user environment
$ tree --dirsfirst -L 4 /cache
/cache
|-- srghma
|   `-- myproject
|       `-- test00000-11
|           `-- cache.zip
`-- test

3 directories, 2 files
$ ls -al ./.mycache || true
ls: ./.mycache: No such file or directory
$ echo "test" > /cache/test
$ mkdir -p ./.mycache
$ echo "test" > ./.mycache/test
$ tree --dirsfirst -L 4 /cache
/cache
|-- srghma
|   `-- myproject
|       `-- test00000-11
|           `-- cache.zip
`-- test

3 directories, 2 files
$ ls -al ./.mycache || true
total 12
drwxr-xr-x    2 root     root          4096 Feb 24 11:45 .
drwxrwxrwx   20 root     root          4096 Feb 24 11:45 ..
-rw-r--r--    1 root     root             5 Feb 24 11:45 test
Creating cache test00000-12...
.mycache: found 2 matching files
No URL provided, cache will be not uploaded to shared cache server. Cache will be stored only locally.
Created cache
Job succeeded

How the gitlab-ci cache is working on docker runner? what is /cache directory? what is cache_dir?

Tags:

gitlab-ci

gitlab-ci-runner

srghma

People also ask

1 Answers

TLDR

Proofs

srghma

Recent Activity

Donate For Us

How the gitlab-ci cache is working on docker runner? what is /cache directory? what is cache_dir?

Tags:

gitlab-ci

gitlab-ci-runner

srghma

People also ask

1 Answers

TLDR

Proofs

srghma

Related questions

Recent Activity

Donate For Us