How to speed up CI build times when using docker?

Question

I currently use docker + travis CI to test/ deploy my app. This works great locally because I have data volumes for things like node_modules etc, and docker's layers provide caching speeding up builds.

However, when I push the code to travis it has to rebuild and install everything from scratch and it takes forever! Travis doesn't support caching docker layers atm. Is there some other way to speed up my builds, or another similar tool that allows docker layer caching?

Answer 1

You might want to investigate how i3wm has solved a similar problem.

The main developer has written on the design behind his Travis CI workflow. Quoting the relevant part:

The basic idea is to build a Docker container based on Debian testing and then run all build/test commands inside that container. Our Dockerfile installs compilers, formatters and other development tools first, then installs all build dependencies for i3 based on the debian/control file, so that we don’t need to duplicate build dependencies for Travis and for Debian.

This solves the immediate issue nicely, but comes at a significant cost: building a Docker container adds quite a bit of wall clock time to a Travis run, and we want to give our contributors quick feedback. The solution to long build times is caching: we can simply upload the Docker container to the Docker Hub and make subsequent builds use the cached version.

We decided to cache the container for a month, or until inputs to the build environment (currently the Dockerfile and debian/control) change. Technically, this is implemented by a little shell script called ha.sh (get it? hash!) which prints the SHA-256 hash of the input files. This hash, appended to the current month, is what we use as tag for the Docker container, e.g. 2016-03-3d453fe1.

See our .travis.yml for how to plug it all together.