How to cache requirements for a Django project on Travis-CI?

Question

As Travis-CI is evolving and extending its feature set it naturally becomes nicer and nicer to use. I recently read this article about "Speeding up the build". A build for the Django project I am working on takes ~25-30 minutes. Almost half of this time is spent on creating the virtualenv, i.e. installing the project's requirements. The other half of the time is used for the actual test run.

There are efforts on cutting down runtime for tests. Yet, I was wondering whether a bigger speed-up was up-for-grabs by caching or bundling the project's requirements. As for Plone there seem to be some options as it uses buildout. I was also looking at WAD. Of course, when caching the requirements they will need to get invalidated upon a requirements update.

Has anyone made any Travis build speed improvements for a (Django) project by cutting down setup time?

Answer 1

Update This is now a first class feature of Travis: http://blog.travis-ci.com/2013-12-05-speed-up-your-builds-cache-your-dependencies/

I've just been playing around with this, and it looks like you can cache the virtualenv site-packages like this (update the path to your python version):

cache:
  directories:
    - /home/travis/virtualenv/python2.7/lib/python2.7/site-packages

There's a little issue that it doesn't cache the bin or the src directories. I tried caching the whole virtualenv directory, but I get strange errors for dependencies installed via git into the src directory.

You are still left with the problem of invalidating old requirements. if you remove something from the requirements, it will persist in the virtualenv so you either have to explicitly remove it with pip (pip remove foo) or wait until Travis create an API to invalidate the cache...

The other option is to use the --download-cache option for pip, then add that directory to the cache:

cache:
  directories:
    - $HOME/.pip-cache/

install:
  - pip install -r requirements.txt --download-cache $HOME/.pip-cache

This will make the downloads faster, but it will still have to compile and install all of the requirements!

Answer 2

This has gotten even easier over the years. The latest way is:

cache: pip

That's it.