Tox 0% coverage

Question

I have a python project where I use:

• pipenv
• tox
• pytest

and many more.

Basically, I want to add tox to my gitlab pipelines. And almost everything seems to work, calling mypy, flake8 or black from tox works fine. But when I call tox -e py37 (so I want to run the tests) with coverage enabled, the tests are run, I can see their results, but the coverage is 0% (100% only on empty __init__ files etc.) and I get the warning: Coverage.py warning: No data was collected. (no-data-collected).

This is my pytest-relevant part of tox.ini:

[tox]
envlist = flake8,mypy,black,py37

[testenv]
extras = tests
commands=
    pytest --cov -ra --tb=short {posargs}

And this is my .coveragerc:

[run]
branch = True
source =
    foo

omit = 
    foo/__init__.py
    foo/__main__.py
    foo/bar/__init__.py
    foo/baz/__init__.py

[report]
exclude_lines =
    pragma: no cover
    if __name__ == .__main__.
show_missing = True

I have a working setup.py which includes all needed packages: pytest, pytest-cov and many more for style etc. (which works). I also have __init__.py in my tests folder.

Funny thing is, if I call the same command I have in tox.ini from my command line (in pipenv): pytest --cov -ra --tb=short, I get the same results but with normal coverage (in my case it's 100%).

Is there a way to fix it? I don't want my pipelines to show wrong coverage + even if they somehow magically would actually get the right one, I still would want to see it locally.

P.S When I was trying to resolve the issue I called tox with tox --sitepackages -e py37 and I got some errors like test command found but not in testenv. And besides those errors, my coverage was fine then. But then I uninstalled tox, pytest and pytest-cov from my global pip register and now with or without --sitepackages flag I still get 0% coverage

Answer 1

The root-cause is that pytest during runnung tox uses the installed sources (under site-packages) while the coverage reporter count the hits files in the working directory (ie. your local git repo).

---

Solution

Add the tox's virtual-env path to coverage:

[testenv]
pytest --cov={envsitepackagesdir}/foo

For more detail read substitutions for virtualenv-related sections

---

Work-around

Set PYTHONPATH to working directory (ie. export PYTHONPATH=.). Note, that you need to pass the PYTHONPATH to tox by:

[testenv]
passenv =
    PYTHONPATH

This is easy, but using this way, you don't test the installation, because the tests runs on the files in the working directory.

Answer 2

Trivial solution

Have usedevelop=True in the tox.ini:

[testenv]
usedevelop=True
pytest --cov=foo

which results in the same behavior as @zvi-baratz's solution but is less hacky.

Caveat

The only caveat here is that if the package has a custom install_command this solution can not be used. Please see the docs for more info.

If that is the case, you can use --cov={envsitepackagesdir}/foo as a work-around.

Why not just use envsitepackagesdir all the time?

Because it results in the coverage report having lengthier paths that point to files in the python environment created by tox instead of the actual project path.