Caching virtual environment for gitlab-ci

Question

I cached Pip packages using a Gitlab CI script, so that's not an issue.

Now I also want to catch a Conda virtual environment, because it reduces time to setup the environment.

I cached a virtual environment. Unfortunately it takes a long time at the end to cache all the venv files.

I tried to cache only the $CI_PROJECT_DIR/myenv/lib/python3.6/site-packages folder and it seems to reduce run time of the pipe.

My question is: am I doing it correctly?

The script is given below:

gitlab-ci.yml

image: continuumio/miniconda3:latest

cache:
  paths:
    - .pip
    - ls -l $CI_PROJECT_DIR/myvenv/lib/python3.6/site-packages
    - $CI_PROJECT_DIR/myvenv/lib/python3.6/site-packages

before_script:
  - chmod +x gitlab-ci.sh
  - ./gitlab-ci.sh

stages:
  - test

test:
  stage: test
  script:
    - python eval.py

gitlab-ci.sh

#!/usr/bin/env bash
ENV_NAME=myenv
ENV_REQUIREMENTS=requirements.txt

if [ ! -d $ENV_NAME ]; then
    echo "Environment $ENV_NAME does not exist. Creating it now!"
    conda create --path --prefix "$CI_PROJECT_DIR/$ENV_NAME"
fi

echo "Activating environment: $CI_PROJECT_DIR/$ENV_NAME"
source activate "$CI_PROJECT_DIR/$ENV_NAME"

echo "Installing PIP"
conda install -y pip

echo "PIP: installing required packages"
echo `which pip`
pip --cache-dir=.pip install -r "$ENV_REQUIREMENTS"

Answer 1

We are successfully using the method outlined in the docs https://docs.gitlab.com/ee/ci/caching/#caching-python-dependencies

# Change pip's cache directory to be inside the project directory since we can
# only cache local items.
variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

# Pip's cache doesn't store the python packages
# https://pip.pypa.io/en/stable/reference/pip_install/#caching
#
# If you want to also cache the installed packages, you have to install
# them in a virtualenv and cache it as well.
cache:
  paths:
    - .cache/pip
    - venv/

There may be other things missing, but your first pass probably misses:

• when reducing size of the whole .../venv/ directory tree, you probably need .../venv/bin as this is required to find the correct python version; see this locally after activateing your venv with the command which -a python3
• if pip will be used again (such as later in your build via some make recipe) you need to move the pip cache as shown above.