What is the correct usage of cache/artifacts in Gitlab CI?

Question

I am facing an issue when cached files are not used in project builds. In my case, I want to download composer dependencies in build stage and then add them into final project folder after all other stages succeeds. I thought that if you set cache attribute into .gitlab-ci.yml file, it will be shared and used in other stages as well. But this sometime works and sometimes not.

Gitlab version is 9.5.4

Here is my .gitlab-ci.yml file:

image: ponk/debian:jessie-ssh

variables:
    WEBSERVER: "[email protected]"
    WEBSERVER_DEPLOY_DIR: "/domains/example.com/web-presentation/deploy/"
    WEBSERVER_CDN_DIR: "/domains/example.com/web-presentation/cdn/"
    TEST_VENDOR: '[ "$(ls -A ${WEBSERVER_DEPLOY_DIR}${CI_COMMIT_REF_NAME}/${CI_COMMIT_SHA}/vendor)" ]'

cache:
  key: $CI_PIPELINE_ID
  untracked: true
  paths:
    - vendor/

before_script:


stages:
    - build
    - tests
    - deploy
    - post-deploy

Build sources:
    image: ponk/php5.6
    stage: build
    script:
        # Install composer dependencies
        - composer -n install --no-progress
    only:
        - tags
        - staging


Deploy to Webserver:
    stage: deploy
    script:
        - echo "DEPLOYING TO ... ${WEBSERVER_DEPLOY_DIR}${CI_COMMIT_REF_NAME}/${CI_COMMIT_SHA}"
        - ssh $WEBSERVER mkdir -p ${WEBSERVER_DEPLOY_DIR}${CI_COMMIT_REF_NAME}/${CI_COMMIT_SHA}
        - rsync -rzha app bin vendor www .htaccess ${WEBSERVER}:${WEBSERVER_DEPLOY_DIR}${CI_COMMIT_REF_NAME}/${CI_COMMIT_SHA}
        - ssh $WEBSERVER '${TEST_VENDOR} && echo "vendor is not empty, build seems ok" || exit 1'
        - ssh $WEBSERVER [ -f ${WEBSERVER_DEPLOY_DIR}${CI_COMMIT_REF_NAME}/${CI_COMMIT_SHA}/vendor/autoload.php ] && echo "vendor/autoload.php exists, build seems ok" || exit 1
        - echo "DEPLOYED"
    only:
        - tags
        - staging

Post Deploy Link PRODUCTION to Webserver:
    stage: post-deploy
    script:
        - echo "BINDING PRODUCTION"
        - ssh $WEBSERVER unlink ${WEBSERVER_DEPLOY_DIR}production-latest || true
        - ssh $WEBSERVER ln -s ${WEBSERVER_DEPLOY_DIR}${CI_COMMIT_REF_NAME}/${CI_COMMIT_SHA} ${WEBSERVER_DEPLOY_DIR}production-latest
        - echo "BOUNDED  $CI_COMMIT_SHA -> production-latest"
        - ssh $WEBSERVER sudo service php5.6-fpm reload
    environment:
        name: production
        url: http://www.example.com
    only:
        - tags

Post Deploy Link STAGING to Webserver:
    stage: post-deploy
    script:
        - echo "BINDING STAGING"
        - ssh $WEBSERVER unlink ${WEBSERVER_DEPLOY_DIR}staging-latest || true
        - ssh $WEBSERVER ln -s ${WEBSERVER_DEPLOY_DIR}${CI_COMMIT_REF_NAME}/${CI_COMMIT_SHA} ${WEBSERVER_DEPLOY_DIR}staging-latest
        - echo "BOUNDED  ${CI_COMMIT_SHA} -> staging-latest"
        - ssh $WEBSERVER sudo service php5.6-fpm reload
    environment:
        name: staging
        url: http://staging.example.com
    only:
        - staging

In Gitlab documentation it says: cache is used to specify a list of files and directories which should be cached between jobs.

From what I understand I've set up cache correctly - I have untracked set to true, path includes vendor folder and key is set to Pipeline ID, which should be the same in other stages as well.

I've seen some set ups which contained Artifacts, but unless you use it with Dependencies, it shouldn't have any effect.

I don't know what I'm doing wrong. I need to download composer dependencies first, so I can copy them via rsync in next stage. Do you have any ideas/solutions? Thanks

Answer 1

Artifacts should be used to permanently make available any files you may need at the end of a pipeline, for example generated binaries, required files for the next stage of the pipeline, coverage reports or maybe even a disk image. But cache should be used to speed up the build process, for example if you compiling a C/C++ binary it usually takes a long time for the first build but subsequent builds are usually faster because it doesn't start from scratch, so if you were to store the temporary files made by the compiler by using cache, it would speed up the compilation across different pipelines.

So to answer you, you should use artifacts because you seem to need to run composer every pipeline but want to pass on the files to the next job. You do not need to explicitly define dependencies in your gitlab-ci.yml because if not defined each job pulls all the artifacts from all previous jobs. Cache should work but it is unreliable and is better for a config where it makes it better but is not a necessity.