Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Caching stack databases between builds in Gitlab CI

I have a large-ish, multi-language project using CMake to build. In this there is a part written in Haskell, and it uses stack to build. The CI builds are handled by Gitlab CI and run in a docker executor. During creation of the docker image ghc is downloaded using stack setup. The Haskell-part of the build still takes a rather long time though, since it downloads and builds all required packages anew for each build.

To cache the local database I added

cache:
  paths:
    - src/utils/.stack-work

(The stack YAML file lives in src/utils/, and the three Haskell packages live in dirs there too.)

This didn't really speed up the build that much though. After reading more carefully about stack databases I realised that snapshots are put into STACK_ROOT (~/.stack). Studying the options for stack suggests that there is no way to specify that snapshots should be stored separately from STACK_ROOT.

Given that Gitlab CI seems to only allow caching of items in the build dir I think I'm left with two options:

  • Use stack --stack-root <folder under build> ... to place the STACK_ROOT in the build dir and then cache the whole thing. This means my cache goes from less than 100M to about 1.6G!
  • Pre-fetch the snapshot we're currently on into the docker image we're using for building.

Now for my questions:

  • Am I correct in thinking that snapshots always are stored in STACK_ROOT?
  • Is there a way to pre-fetch a specific snapshot, similar to how stack setup fetches ghc?

Or maybe I'm completely wrong, and there's a better route towards limiting the cache size.

like image 417
Magnus Avatar asked Nov 08 '22 13:11

Magnus


1 Answers

After @dsign's comment and a bit of thinking I've come to a solution that is satisfactory:

  • Yes, pre-fetching by (sort of) building the software once during docker image build is all right.
  • The way to pre-fetch is by running stack install --only-dependencies.
like image 177
Magnus Avatar answered Nov 15 '22 08:11

Magnus