How do you use pipenv in a GitHub action?

Question

I'm using pipenv to install python dependencies for a GitHub action. This is what my pipfile looks like:

name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
pylint = "*"

[packages]
pandas = "*"
pygithub = "*"
matplotlib = "*"

[requires]
python_version = "3.8"

This is what the Dockerfile looks like:

FROM python:3

COPY main.py /
COPY Pipfile /
COPY Pipfile.lock /
COPY views.csv /

# https://github.com/pypa/pipenv/issues/4273
RUN pip install 'pipenv==2018.11.26'
RUN pipenv install --deploy --ignore-pipfile

ENTRYPOINT ["pipenv", "run", "python", "./main.py"

I can run the docker image locally on my machine and everything works as expected, but when I push it to GitHub, it fails with:

Virtualenv location: /github/home/.local/share/virtualenvs/workspace-0SZQOxG8
Traceback (most recent call last):
  File "./main.py", line 1, in <module>
    from github import Github
ModuleNotFoundError: No module named 'github'

For some reason, when the docker image is run on the GitHub vm, it's not picking up any of the dependencies.

Edit 7/28/2020:

This is the main.yml for the workflow that triggers the action.

on: 
  push: 
    branches: 
      - master


# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "insights_job"
  insights_job:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
    # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
    - uses: actions/checkout@v2
      
    - name: GitHub insights 
      id: insights
      uses: ./

Answer 1

Short version

Just add this to the top of the Dockerfile:

# Tells pipenv to create virtualenvs in /root rather than $HOME/.local/share.
# We do this because GitHub modifies the HOME variable between `docker build` and
# `docker run`
ENV WORKON_HOME /root

# Tells pipenv to use this specific Pipfile rather than the Pipfile in the 
# current working directory (the working directory changes between `docker build` 
# and `docker run`, this ensures we always use the same Pipfile)
ENV PIPENV_PIPFILE /Pipfile

Long version

As can be seen here, Github actions defines the HOME environment variable to be /github/home.

We can see that this variable is then passed to to the docker container via the run command that is visible in the action's step's logs:

/usr/bin/docker run --name c201c6ad6ba986775dbb96d3c072d294f3c8_a057c6 --label 87c201 --workdir /github/workspace --rm -e HOME -e GITHUB_JOB -e GITHUB_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_REPOSITORY_OWNER -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_ACTOR -e GITHUB_WORKFLOW -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GITHUB_EVENT_NAME -e GITHUB_SERVER_URL -e GITHUB_API_URL -e GITHUB_GRAPHQL_URL -e GITHUB_WORKSPACE -e GITHUB_ACTION -e GITHUB_EVENT_PATH -e RUNNER_OS -e RUNNER_TOOL_CACHE -e RUNNER_TEMP -e RUNNER_WORKSPACE -e ACTIONS_RUNTIME_URL -e ACTIONS_RUNTIME_TOKEN -e ACTIONS_CACHE_URL -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/j/j":"/github/workspace" 87c201:c6ad6ba986775dbb96d3c072d294f3c8

(Notice the -e home flag)

pipenv install creates a virtualenv in $HOME/.local/share/virtualenvs and installs all the dependencies in that directory. pipenv run ... then uses that same path find the virtualenv to run the required command.

During the build stage of the Docker image (docker build), $HOME has the default value of /root (because that's the root user's home directory), so virtualenv gets created in /root/.local/share/virtualenvs.

However, during the docker run stage, HOME is set to /github/home (as described above). Thus, the virtualenv gets created in /github/home/.local/share/virtualenvs.

To summarize -

pipenv install uses /root/.local/share/virtualenvs

pipenv run uses /github/home/.local/share/virtualenvs

That means the dependencies are installed in one placed but expected to be in a different place when the script is actually ran.

To overcome this, we can use set the WORKON_HOME in the Dockerfile using an ENV statement.

The WORKON_HOME variable is respected by pipenv and it will prefer to place its virtual environments in the path defined in WORKON_HOME rather than the path defined in HOME.

So the solution would be to add

ENV WORKON_HOME /root

To the Dockerfile.

However, this is not enough, since pipenv chooses the virtualenv according to the location of the pipenv file. The --workdir is set to /github/workspace by Github and then the entire project source code is mounted into that folder, so when we pipenv run we're actually using the Pipenv file in /github/workspace. This is a different Pipenv file than the one that was copied during the docker build stage into / and that was used for the pipenv install. Since two different Pipenv locations are used, two different virtualenvs will be used.

To solve that we can use the PIPENV_PIPEFILE environment variable. This variable tells pipenv to look for the Pipfile in the location specified by that variable instead of in the current working directory.

Let's also add this to our Dockerfile:

ENV PIPENV_PIPFILE /Pipfile