coverage.py does not cover script if py.test executes it from another directory

Question

I got a python script which takes command line arguments, working with some files. I'm writing succeeding tests with py.test putting this script through its paces, executing it with subprocess.call.

Now I want to analyze code coverage with coverage.py. Coverage, when used via the pytest-cov plugin (which has subprocess-handling built-in), does not see/cover my script when it is called from a temporary testing directory created with py.test's tmpdir fixture. Coverage does see my script when it's called in the directory it resides in (and the filename argument points to a remote path).

In both situations, my tests pass! Coverage 3.6, pytest-2.3.5, pytest-cov 1.6, all from PyPi.

Question: How can I get coverage to recognize my script even if it's executed in another directory? Is this a bug in coverage, or something which is just not possible to do? Would be surprised if the latter, after all, tmpdir is a stock mechanism of py.test...

Minimal example:

I got a script my_script.py which just echoes the contents of a file arg_file.txt supplied via command-line argument. In two different tests, this is once called in a tmpdir, and once in the script's location. Both tests pass, but the in the tmpdir test, I get no coverage information!

Test run:

~/pytest_experiment$ py.test -s =================================== test session starts ==================================== platform linux2 -- Python 2.7.4 -- pytest-2.3.5 plugins: cov collected 2 items   tests/test_in_scriptdir.py  set_up: In directory /tmp/pytest-52/test_10 Running in directory /home/cbuchner/pytest_experiment Command: ./my_script.py /tmp/pytest-52/test_10/arg_file.txt --Contents of arg_file.txt--  . tests/test_in_tmpdir.py  set_up: In directory /tmp/pytest-52/test_11 Running in directory /tmp/pytest-52/test_11 Command: /home/cbuchner/pytest_experiment/my_script.py arg_file.txt --Contents of arg_file.txt--  .  ================================= 2 passed in 0.06 seconds ================================= 

Coverage:

~/pytest_experiment$ py.test --cov=my_script.py tests/test_in_scriptdir.py=================================== test session starts ==================================== platform linux2 -- Python 2.7.4 -- pytest-2.3.5 plugins: cov collected 1 items   tests/test_in_scriptdir.py . --------------------- coverage: platform linux2, python 2.7.4-final-0 ---------------------- Name        Stmts   Miss  Cover ------------------------------- my_script       3      0   100%  ================================= 1 passed in 0.09 seconds ================================= ~/pytest_experiment$ py.test --cov=my_script.py tests/test_in_tmpdir.py=================================== test session starts ==================================== platform linux2 -- Python 2.7.4 -- pytest-2.3.5 plugins: cov collected 1 items   tests/test_in_tmpdir.py .Coverage.py warning: No data was collected.  --------------------- coverage: platform linux2, python 2.7.4-final-0 ---------------------- Name    Stmts   Miss  Cover ---------------------------  ================================= 1 passed in 0.09 seconds ================================= 

The files are here: https://gist.github.com/bilderbuchi/6412754

Edit: Interstingly, when running the coverage tests with -s, too, there's more curious output - coverage warns that No data was collected, when obviously it was collected, and in the tmpdir test warns that Module my_script.py was never imported.??

~/pytest_experiment$ py.test -s --cov=my_script.py tests/test_in_scriptdir.py =================================== test session starts ==================================== platform linux2 -- Python 2.7.4 -- pytest-2.3.5 plugins: cov collected 1 items   tests/test_in_scriptdir.py  set_up: In directory /tmp/pytest-63/test_10 Running in directory /home/cbuchner/pytest_experiment Command: ./my_script.py /tmp/pytest-63/test_10/arg_file.txt --Contents of arg_file.txt--  Coverage.py warning: No data was collected. . --------------------- coverage: platform linux2, python 2.7.4-final-0 ---------------------- Name        Stmts   Miss  Cover ------------------------------- my_script       3      0   100%  ================================= 1 passed in 0.09 seconds ================================= ~/pytest_experiment$ py.test -s --cov=my_script.py tests/test_in_tmpdir.py=================================== test session starts ==================================== platform linux2 -- Python 2.7.4 -- pytest-2.3.5 plugins: cov collected 1 items   tests/test_in_tmpdir.py  set_up: In directory /tmp/pytest-64/test_10 Running in directory /tmp/pytest-64/test_10 Command: /home/cbuchner/pytest_experiment/my_script.py arg_file.txt --Contents of arg_file.txt--  Coverage.py warning: Module my_script.py was never imported. Coverage.py warning: No data was collected. Coverage.py warning: Module my_script.py was never imported. Coverage.py warning: No data was collected. .Coverage.py warning: No data was collected.  --------------------- coverage: platform linux2, python 2.7.4-final-0 ---------------------- Name    Stmts   Miss  Cover ---------------------------  ================================= 1 passed in 0.09 seconds ================================= 

Answer 1

I encountered the same issue when calling "py.test --cov ..." from tox. I found a hint on this page: http://blog.ionelmc.ro/2014/05/25/python-packaging/ even though it does not mention this explicitly. Using "--develop" for tox will make sure that coverage data gathering is called from the same directory as coverage analysis. This section in tox.ini made it work for me to have a test environment for coverage:

[tox] envlist = ...,py34,cov  [testenv:cov] # necessary to make cov find the .coverage file # see http://blog.ionelmc.ro/2014/05/25/python-packaging/ usedevelop = true commands = py.test --cov=<MODULE_NAME> deps = pytest pytest-cov