Is it possible to rebuild only updated files in Gitlab CI?

Question

I'm using this script for my Gitlab CI build stage (only relevant part is shown):

cache:
  key: "$CI_BUILD_REF"
  paths:
    - bin/
    - build/

build:
  image: <my_build_image>
  stage: build
  script:
    - "make PLATFORM='x86_64-linux-gnu' BUILD='release' JOBS=8 all"
  only:
    - master
    - tags
    - merge-requests
  artifacts:
    untracked: true
    paths:
      - bin/x86_64-linux-gnu/release

I thought what if I'll add bin and build dirs into the cache, make won't rebuild the whole project every time (just like it behaves locally), but it seems what CI runner overwrites my src dir every time, so timestamps on the files is being updated too and make think each file is updated. I thought about including src dir into the cache, but it's included in the repo and I'm not sure this is correct. So, which is the best way to rebuild gitlab ci project using previously built binaries?

Answer 1

I see you are using $CI_BUILD_REF as a cache key; although this variable is deprecated, it seems to work and provides the commit's SHA1. Is that really what you intended, to create separate caches per commit (not even per branch)? So for any new commit there wouldn't be a cache anyways?

I'd probably even use a static cache key in order to maximize caching (while using minimal cache storage), or maybe per branch.

Maybe also the Git checkouts and/or branch switches touch the source files too often. I have implemented a similar strategy in one of my projects, but there I have a distinct "cached" folder to where I /rsync/ the files from the checkout.

The shared runners of Gitlab.com do seem to leave the file modification time intact when using a cache, and even on the main checkout.

I've put up a sample project with a CI job that demonstrates the fac tat https://gitlab.com/hannibal218bc/test-build-cache-xtimes/-/jobs/1022404894 :

• the job stats the directory's contents
• creates a cached directory if it not yet exists
• copies the README.md file
• "compiles" the file to a README.built file.

As you can see in the output, the modification timestamp of the README.built is the runtime from the previous job:

$ cd cached
$ stat README.* || true
  File: README.built
  Size: 146         Blocks: 16         IO Block: 4096   regular file
Device: 809h/2057d  Inode: 2101510     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-02-10 23:06:13.000000000 +0000
Modify: 2021-02-10 23:02:39.000000000 +0000   <<< timestamp from previous job
Change: 2021-02-10 23:06:13.000000000 +0000