GitLab CI build fails with error: no space left on device.

Question

Here is the scenario:

• created a kubernetes cluster in a google cloud project with default settings
• configured auto dev ops on gitlab as shown in the quickstart
• configured a .gitlab-ci.yml file as shown in the default kubernetes template (only changed KUBE_DOMAIN with my custom domain after setting correct DNS)
• once I push something to master branch, a build job is triggered in the pipeline buit it fails. Here follows the console output:

---

    Running with gitlab-runner 10.4.0-rc1 (fb4078b3)
  on docker-auto-scale (fa6cab46)
Using Docker executor with image registry.gitlab.com/gitlab-examples/kubernetes-deploy ...
Using docker image sha256:f5f6ae0793e2d8907a16c95a4062bc93a98de76e5caf476afdd75a55f0a4808c for predefined container...
Pulling docker image registry.gitlab.com/gitlab-examples/kubernetes-deploy ...
Using docker image registry.gitlab.com/gitlab-examples/kubernetes-deploy ID=sha256:94b90ce9f86870a6ad41b6696878dcd3f3053a34547e55acc90de6c6bdc09735 for build container...
Running on runner-fa6cab46-project-5122416-concurrent-0 via runner-fa6cab46-srm-1515712084-0dfb4a6e...
Cloning repository...
Cloning into '/builds/Cardamone/gulp-bootstrap-kickstart'...
Checking out 64575330 as master...
Skipping Git submodules setup
$ command build
Checking docker engine...
Missing docker engine to build images.
Running docker:dind locally with graph driver pointing to '/cache/docker'
Waiting for docker...
time="2018-01-11T23:09:41Z" level=warning msg="The \"-g / --graph\" flag is deprecated. Please use \"--data-root\" instead" 
time="2018-01-11T23:09:41.376948630Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found" 
time="2018-01-11T23:09:41.378177859Z" level=info msg="libcontainerd: new containerd process, pid: 27" 
time="2018-01-11T23:09:41.395771265Z" level=warning msg="failed to rename /cache/docker/tmp for background deletion: rename /cache/docker/tmp /cache/docker/tmp-old: no such file or directory. Deleting synchronously" 
time="2018-01-11T23:09:41.431476805Z" level=info msg="Graph migration to content-addressability took 0.00 seconds" 
time="2018-01-11T23:09:41.432582392Z" level=info msg="Loading containers: start." 
time="2018-01-11T23:09:41.433722690Z" level=warning msg="Running modprobe bridge br_netfilter failed with message: modprobe: can't change directory to '/lib/modules': No such file or directory\n, error: exit status 1" 
time="2018-01-11T23:09:41.434401118Z" level=warning msg="Running modprobe nf_nat failed with message: `modprobe: can't change directory to '/lib/modules': No such file or directory`, error: exit status 1" 
time="2018-01-11T23:09:41.435034223Z" level=warning msg="Running modprobe xt_conntrack failed with message: `modprobe: can't change directory to '/lib/modules': No such file or directory`, error: exit status 1" 
time="2018-01-11T23:09:41.513018523Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.18.0.0/16. Daemon option --bip can be used to set a preferred IP address" 
time="2018-01-11T23:09:41.545527019Z" level=info msg="Loading containers: done." 
time="2018-01-11T23:09:41.573837463Z" level=info msg="Daemon has completed initialization" 
time="2018-01-11T23:09:41.573958530Z" level=info msg="Docker daemon" commit=02c1d87 graphdriver=overlay2 version=17.06.0-ce 
time="2018-01-11T23:09:41.593210249Z" level=info msg="API listen on /var/run/docker.sock" 

Building application...
Building Heroku-based application using gliderlabs/herokuish docker image...
Unable to find image 'gliderlabs/herokuish:latest' locally
latest: Pulling from gliderlabs/herokuish
c02c7df4a131: Pulling fs layer
a3ed95caeb02: Pulling fs layer
ea92c993fbb7: Pulling fs layer
a3b5eef14734: Pulling fs layer
0e3c369b6f53: Pulling fs layer
9b9a3918b638: Pulling fs layer
848252ff228c: Pulling fs layer
df3273c9ccc6: Pulling fs layer
a3b5eef14734: Waiting
0e3c369b6f53: Waiting
9b9a3918b638: Waiting
848252ff228c: Waiting
df3273c9ccc6: Waiting
ea92c993fbb7: Verifying Checksum
ea92c993fbb7: Download complete
a3ed95caeb02: Verifying Checksum
a3ed95caeb02: Download complete
0e3c369b6f53: Verifying Checksum
0e3c369b6f53: Download complete
c02c7df4a131: Verifying Checksum
c02c7df4a131: Download complete
9b9a3918b638: Verifying Checksum
9b9a3918b638: Download complete
848252ff228c: Verifying Checksum
848252ff228c: Download complete
c02c7df4a131: Pull complete
a3ed95caeb02: Pull complete
ea92c993fbb7: Pull complete
df3273c9ccc6: Verifying Checksum
df3273c9ccc6: Download complete
a3b5eef14734: Verifying Checksum
a3b5eef14734: Download complete
time="2018-01-11T23:10:38.379477370Z" level=error msg="Not continuing with pull after error: failed to register layer: Error processing tar file(exit status 1): write /usr/share/terminfo/n/ncr7900iv: no space left on device" 
time="2018-01-11T23:10:38.380896411Z" level=info msg="Layer sha256:2381de6cd1138e16c1f8c55d20f32bfb251442371409c8390f0c0ae7201f0e2f cleaned up" 
docker: failed to register layer: Error processing tar file(exit status 1): write /usr/share/terminfo/n/ncr7900iv: no space left on device.
See 'docker run --help'.
ERROR: Job failed: exit code 1

maybe I missed something in the configuration of clusters/project in google platform? do I need to "add space in some way" to the app?

thanks for any suggestion

Answer 1

I wasn't able to reproduce the error you're seeing. My steps:

• Went through the steps in the AutoDevOps quickstart on a new Gitlab project
• (Used the Ruby on Rails template)
• Created a Kubernetes cluster via the Gitlab UI -- used default Node machine type: n1-standard-2 (2 vCPUs, 7.5 GB memory) (CPU and memory aren't related to being out of disk space, though.) Each node has a 100GB boot disk ("standard persistent disk.")
• Enabled AutoDevops
• Deleted the repo's Dockerfile so that GitLab would default to Heroku buildpacks (to pull the Docker image that threw your error)
• The pipeline ran. And the build stage was able to fully download the image:

Digest: sha256:1de05c84afa3abf1df66efca6a928d33ea142cd564b31c7e33d09c8104d8534a
Status: Downloaded newer image for gliderlabs/herokuish:latest

Ideas on why you're seeing that "out of space" error: gliderlabs/herokuish:latest is a 1.21GB Docker image. And the Build stage itself is running "docker in docker" (A Kubernetes pod, which is a docker container, is trying to execute its docker pull command.)

While it seems (from your df -h) that the gitlab runner Pod has enough disk space total, it's possible Docker is writing to a temp filesystem and running out of space. One solution could be to switch Docker's storage driver from VFS to Overlay, in your .gitlab_ci.yml:

variables:
  DOCKER_DRIVER: overlay2

Sources: https://docs.gitlab.com/ce/ci/docker/using_docker_build.html#using-the-overlayfs-driver https://gitlab.com/gitlab-com/support-forum/issues/1025