Docker volume cleanup of gitlab-runner

Question

I am new to docker and jenkins. However ultimately I am creating a job in jenkins so that I can delete the volume cache of gitlab-runner which is stored in our linux machines (CentOS7)

To achieve this, I am creating a periodic job every 6 hours with the following command in jenkins:

docker volume prune -f 

However it doesn't clean up the space at all. This is the output of jenkins job :

Started by timer
Running as SYSTEM
Building remotely on buildbng17 (gitlab) in workspace /mnt/data0/jenkins/workspace/gitlab-cleanup
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] Done
[gitlab-cleanup] $ /bin/sh -xe /tmp/jenkins3799570002967825583.sh
+ find /mnt/data0/gitlab/data/backups/ -name '*.tgz' -mtime +30
/mnt/data0/gitlab/data/backups/etc-gitlab-1611415968.tgz
Started calculate disk usage of build
Finished Calculation of disk usage of build in 0 seconds
Started calculate disk usage of workspace
Finished Calculation of disk usage of workspace in 0 seconds
Finished: SUCCESS

When I went to my buildbng17 machine and checked if there is any volume their, so there was volume which I had to clean up by performing

docker volume prune

The only drawback is that I need to do it manually and give that "y" when it asks for confirmation. And the data gets clean.

What command should I give in jenkins so that it will automatically clean up the volume without asking for confirmation? (because docker volume prune -f is not working)

I even tried to run the docker volume prune -f command manually in my linux machine, it still doesn't clean up the volume and shows 0kb freed up (However there is the volume which I checked from docker volume ls)

Is there any better approach we can do to automate it inside gitlab-runner itself ? Or some better feature of jenkins or docker of which I am not aware of?

The gitlab-runner keeps on running and covers the diskspace.

One more thing : We are using Gitlab and Gitlab-runner on docker.

Thanks in advance, Sameer

Answer 1

Run these commands to do cleanup:

# Remove exited containers
docker ps -a -q -f status=exited | xargs --no-run-if-empty docker rm -v

# Remove dangling images
docker images -f "dangling=true" -q | xargs --no-run-if-empty docker rmi

# Remove unused images
docker images | awk '/ago/  { print $3}' | xargs --no-run-if-empty docker rmi

# Remove dangling volumes
docker volume ls -qf dangling=true | xargs --no-run-if-empty docker volume rm

Answer 2

GitLab Runner provides a clear-docker-cache script. It is recommended to run it weekly via cron.

Create /etc/cron.weekly/clear-docker-cache with the following contents:

#!/bin/sh
/usr/share/gitlab-runner/clear-docker-cache

Running /usr/share/gitlab-runner/clear-docker-cache will default to running clear-docker-cache prune-volumes, which basically runs docker system prune --volumes -af.

If you don't want to prune volumes, use /usr/share/gitlab-runner/clear-docker-cache prune.

If you want to add a filter, add the --filter parameter, like --filter "until=24h" to prune prior to the last 24 hours.

If you just want to see what space could be reclaimed, run /usr/share/gitlab-runner/clear-docker-cache space.

---

Another option is to use a pre-build script. GitLab CI/CD runner clean-up with pre-build scripts. It's possible to use both methods.