Cache Brew builds with travis ci

Question

I have a Travis CI osx build with a brew dependency that has to be built from source.

I know that Travis has the cache feature, but it doesn't have any documentation on how to cache brew builds or outputs.

Any idea on how to cache a brew package in travis?

Answer 1

There are 3 separate, loosely related problems here:

• Cache downloaded bottles
• Cache locally-built bottles
• Cache Homebrew metadata

You don't necessarily need all three, so follow whichever sections fit your needs.

---

Cache downloaded bottles

• Add $HOME/Library/Caches/Homebrew to Travis' cache _{(actually, this path is supposed to be retrieved with brew --cache but you can't call it here, can you)}

  cache:   directories:     - $HOME/Library/Caches/Homebrew 

• Run brew cleanup at before_cache stage -- otherwise, the cache will grow indefinitely as new package versions are released

  before_cache:   - brew cleanup 

Cache locally-built bottles

The full code is too long to list it here so giving the algorithm.

This is in addition to the previous section. If using without it, save local bottles somewhere outside of Homebrew cache at on installing step and add them to the cache under appropriate names at the at startup step below.

• On installing:

  • Check package's dependencies with brew deps recursively
    • If a bottle for the package is not available for your environment (no (bottled) in brew info <pkg> output), include build dependencies with --include-build

  • For each of the packages and dependencies,

    • If it's already installed (brew list --versions <pkg> succeeds) and latest version (absent from brew outdated), skip it
    • If an older version is present, in the following steps, you'll need to install the new version alongside the old one:
      • brew unlink the old version if it's not keg-only (no [keg-only] in brew info output)
      • Invoke all brew install's with --force
    • If a bottle is available, just brew install it

    • If a bottle is not available,

      • Build and install it with the following sequence:

        brew install --build-bottle <pkg> brew bottle --json <pkg> brew uninstall --ignore-dependencies <pkg> brew install <bottle> 

        _{(There doesn't seem to be any official way to get the names of the resulting bottle and JSON file. I took the bottle name from brew bottle output and inferred JSON file name from it.)}

      • Add the bottle info into the package's formula

        brew bottle --merge --write <json file> 

      • Save the bottle file into Travis cache under an appropriate name given by brew --cache <pkg>

        • _{Only do this after adding bottle info -- otherwise, you'll get a path to the source package instead.}
        • _{(Homebrew also makes symlinks to downloaded files in $HOME/Library/Caches/Homebrew. You don't need to do this.)}
      • Save the JSON file for later use. Make sure to add its location to Travis cache.

• At startup:

  • Do brew update if you're going to
  • Go through saved .json files. For each one, check if the local bottle is still appropriate (by comparing versions and rebuild numbers; you can parse the output of brew info --json=v1 <pkg> and brew info --json=v1 <bottle> for this data).
    • Delete the cached bottle and the .json if not
      • _{Since you won't be able to get the path to your bottle with brew --cache at this point, you need to have saved it independently. Symlinks aren't being saved in Travis' cache as of this writing, so I ended up using regular files that held the paths.}
    • Re-add the bottle info into the formula like above if yes
      • _{There's also an unlikely possibility that they change the download URL in the formula without bumping the version -- then the bottle's expected cached name will change because the hash in it is the hash of the download URL. To allow for this, check if brew --cache <pkg> still points to your bottle after adding the info.}

• At before_cache:

  • If you're using brew cleanup from the previous section, save your locally-built bottle files from cache somewhere before running it because cleanup may delete those that weren't needed this time around. After cleanup, restore those that were deleted.

Cache Homebrew metadata

^{(Again, the full code is too long so giving the algorithm.)}
If you run brew update --verbose _{(and make sure there are no secret variables in .travis.yml or your Travis project settings -- brew prints many status messages only if stdout is a tty)} -- you'll see what exactly consititutes a Homebrew selfupdate operation -- thus what you should cache:

• Pulling (actually, rebase'ing by default) into a few paths that are actually git repositories:
  • /usr/local/Homebrew -- Homebrew itself
  • /usr/local/Homebrew/Library/Taps/*/* -- installed taps
• Going through the taps and cache and migrating obsolete bits. Since Travis cache content is added to existing directory structure rather than replaces it, the 2nd time around, there may be strange actions and errors caused by files that were deleted as part of update, but are there again in the new VM. The ones I witnessed:
  • _{will always try migrating Taps/caskroom/homebrew-cask to Taps/homebrew/homebrew-cask, creating a copy at Taps/homebrew/homebrew-cask/homebrew-cask. If cached, this copy will cause an "error: file exists" on the next run.}
  • will always try to import lots of non-committed files into Taps/homebrew/homebrew-versions

So, the actions will be:

• Add /usr/local/Homebrew to Travis cache

  • adding /usr/local/Cellar and /usr/local/opt proved to be a bad idea: first, they are too large, causing a timeout while making and uploading cache; second, this is unsafe 'cuz postinstall scripts may affect other arbitrary parts of the system, so one should install new package versions from (cached) bottles each time rather than cache the result. Installing a bottle only takes a few seconds anyway.

• Before brew update: clean up Homebrew codebase

  • Delete the Taps/caskroom/homebrew-cask dir if Taps/homebrew/homebrew-cask exists
  • Find all git repos under /usr/local/Homebrew (find -type d -name .git, get dirname of the result) and run git clean -fxd in each to get rid of Travis' leftovers
  • Clean up Homebrew cache from leftovers, too, with brew cleanup (if using in conjunction with the previous section, see there for additional operations) -- otherwise, you'll get lots of errors in brew update on the "Migrating cache entries..." stage.

• At brew update:

  • Use brew update --merge instead -- it will auto-resolve any possible conflicts with your local commits with bottle info

• When re-adding local bottles (if using in conjunction with the previous section):

  • Don't re-add bottle info into the formula if it's already present there
  • If package version has changed and your bottle info is present in the formula, remove it from the formula and git commit the result. There's no stock way to do that, so you'll have to parse and edit the formula file with a script and delete the corresponsing line from bottle do table. The path to the formula file is retrieved with brew formula <pkg>.

• When installing:

  • If using 3rd-party taps, always check if you already have that tap installed:

    brew tap | grep -qxF <tap> || brew tap <tap> 

    • Since symlinks aren't saved in Travis cache, pins will likely not be remembered. But it won't hurt to check for them, too:

      brew tap --list-pinned | grep -qxF <tap> || brew tap-pin <tap> 

• At before_cache:

  • Delete Taps/homebrew/homebrew-cask/homebrew-cask if it exists