No space on device with Jenkins and Docker - how to cleanup properly

Question

We're running Jenkins (version 2.60.1) on an Ubuntu 16.04.1 server. One of the issues we've been running into recently is that we routinely get the error "no space left on device".

I understand when using Docker there needs to be a strict clean-up process due to the files that are left behind and taking up unnecessary space.

We're using the CloudBees Docker Build and Publish plugin to handle the build and push to AWS ECS. I thought about removing all the unused Images. The thing is if I login to the Jenkins instance (over SSH) and try to run the docker command it gives - "Cannot connect to the Docker daemon. Is the docker daemon running on this host?"

I suppose somehow I need to do this from within Jenkins environment or part of the plugin?

Anyone dealt with this before or have some advice? - I'd really appreciate it.

Answer 1

in order to get past the "cannot connect to the docker docker daemon" issue, figure out what users are in the docker group

grep 'docker' /etc/group

and then run the docker cleanup commands (you'll want to turn it into a script you run on cron or something) as one of those users. or get sudo access with another user and use sudo:

sudo docker rmi [image_name_here]

here's the contents of an example cleanup script (/usr/local/bin/clean_up_docker_stuff_on_ci_agent or similar):

#!/bin/bash

# stop containers that have been running for more than a day (may not be valid in your context if you intend run things for a long time)
docker ps -a | egrep " days" | awk '{print $1}' | grep -v CONTAINER | xargs docker stop

# remove all exited containers
docker ps -a | egrep "Exited|Created" | awk '{print $1}' | grep -v CONTAINER | xargs docker rm

# remove old images
docker images | egrep 'weeks|months' | awk '{print $1 ":" $2}' | xargs docker rmi -f
docker images | egrep 'weeks|months' | grep '<none>' | awk '{ print $3 }' | xargs docker rmi -f

# kill stray volumes
docker volume ls -qf dangling=true | xargs -r docker volume rm

As Szymon Stepniak mentions in his answer, if you're using docker >= 1.13, there are simpler options.

cron example (20 after every hour):

20 * * * * /usr/local/bin/clean_up_docker_stuff_on_ci_agent > /dev/null 2>&1

Answer 2

After using the script provided by burnettk below it seemed that while some space was freed after time running more builds I was back at the same place, no space on my EBS volume. It simply does not make sense that I would have to add more storage and pay AWS even more on my monthly bill.

In doing some investigation I discovered that for EACH build there were approximately 7 images created (docker images -a) consisting of about 1.4GB each, ie 9GB/build. The first 2 are tagged with the build # and latest while the rest are tagged .

It's really not important that all these images are stored on this server as the purpose is for build and anyway they are pushed to ECR. So I've added the following into my script so that only the latest docker image is kept:

docker rmi $(docker images | sed 1,3d | awk '{print $3}')

Lastly, I have also adjusted my docker build command by adding the --rm argument so that it will remove intermediate containers after building.

docker build --rm

Hope this is helpful!