Couldn't use data file .coverage: unable to open database file

Question

A strange issue with permissions occured when pushing to GitHub. I have a test job which runs tests with coverage and then pushes results to codecov on every push and pull request. However, this scenario only works with root user.

If running with digitalshop user it throws an error:

Couldn't use data file '/digital-shop-app/.coverage': unable to open database file

My question is: how to run coverage in docker container so it won't throw this error? My guess is that it's because of permissions.

docker-compose.yml:

version: '3.9'

services:
  test:
    build: .
    command: >
      sh -c "
        python manage.py wait_for_db &&
        coverage run --source='.' manage.py test mainapp.tests &&
        coverage report &&
        coverage xml
      "
    volumes: 
      - ./digital-shop-app:/digital-shop-app
    env_file: .env
    depends_on: 
      - db

  db:
    image: postgres:13-alpine
    environment:
      - POSTGRES_DB=${DB_NAME}
      - POSTGRES_USER=${DB_USER}
      - POSTGRES_PASSWORD=${DB_PASS}

Dockerfile:

FROM python:3.9-alpine3.13

ENV PYTHONUNBUFFERED 1

COPY ./requirements.txt /requirements.txt
COPY ./digital-shop-app /digital-shop-app
COPY ./scripts /scripts

WORKDIR /digital-shop-app

RUN python -m venv /py && \
    /py/bin/pip install --upgrade pip && \
    apk add --no-cache bash && \
    apk add --update --no-cache postgresql-client && \
    apk add --update --no-cache --virtual .tmp-deps \
        build-base jpeg-dev postgresql-dev musl-dev linux-headers \
        zlib-dev libffi-dev openssl-dev python3-dev cargo && \
    apk add --update --no-cache libjpeg && \
    /py/bin/pip install -r /requirements.txt && \
    apk del .tmp-deps && \
    adduser --disabled-password --no-create-home digitalshop && \
    chown -R digitalshop:digitalshop /py/lib/python3.9/site-packages && \
    chmod -R +x /scripts

ENV PATH="/scripts:/py/bin:/py/lib:$PATH"

USER digitalshop

CMD ["run.sh"]

Answer 1

Problem

I ran into the same error when running pytest with coverage using docker-compose in the github-hosted ubuntu-latest image in GitHub Actions. This is an instance of the Docker host file system owner matching problem.

In short, the user on the host (the github action runner) and the user in on the container (where my pytest suite runs) have different UIDs. The mounted directory app is owned by the user on the host. When the user in the container attempts to write to the app/.coverage, permission is denied (since this user is not the owner).

The fix

In my case, I solved the issue by matching the UID of my docker image's default user with that of the github actions runner user, 1001. I added this to my Dockerfile to accomplish this:

# Make the default user have the same UID as the github actions "runner" user.
# This to avoid permission issues when mounting volumes.
USER root
RUN usermod --uid 1001 <image_default_user>
USER <image_default_user>

All relevant files

app/test_utils/docker-compose.yml:

version: "3.9"

services:
  app:
    build:
      context: ../
      dockerfile: ./test_utils/Dockerfile
    container_name: app
    volumes:
      - ..:/app

app/test_utils/Dockerfile:

FROM <my base image>

# Make the default user have the same UID as the github actions "runner" user.
# This to avoid permission issues when mounting volumes, see
USER root
RUN usermod --uid 1001 <image_default_user>
USER <image_default_user>

COPY . /app
WORKDIR /app

RUN pip3 install -r requirements.txt -r requirements_test.txt

app/.github/unittests.yml:

name: Run unit tests, Report coverage

on:
  pull_request:
    paths:
      - app/*
      - .github/workflows/unittests.yml

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - name: Checks out the repo
        uses: actions/checkout@v2

      - name: Build docker image
        run: docker-compose -f test_utils/docker-compose.yml build

      - name: Run unit tests & produce coverage report
        # Adapted from the docker example in
        # https://github.com/MishaKav/pytest-coverage-comment?tab=readme-ov-file#example-usage
        run: |
          docker-compose \
            -f test_utils/docker-compose.yml \
            run app \
            pytest \
              --cov-report=term-missing:skip-covered \
              --junitxml=/app/pytest.xml \
              --cov=/app \
              /app \
            | tee pytest-coverage.txt

      - name: Pytest coverage comment
        uses: MishaKav/pytest-coverage-comment@main
        with:
          pytest-coverage-path: pytest-coverage.txt
          junitxml-path: pytest.xml