Pytest and coverage: why do coverage results vary with directory structure?

Question

I have a working test suite using Pytest on a fairly large django project. Problem is I am unable to achieve proper results using coverage, and I am wondering if it may be because of the projects directory structure.

Consider the following sample of the directory tree:

.
├── apps
│   ├── api
│   │   ├── __init__.py
│   │   ├── tests
│   │   │   ├── __init__.py
│   │   │   └── views
│   │   │       ├── __init__.py
│   │   │       └── test_tickets.py
│   │   └── views
│   │       ├── __init__.py
│   │       ├── tickets.py
│   ├── support
│   │   ├── __init__.py
│   │   ├── tests
│   │   │   ├── __init__.py
│   │   │   └── utils
│   │   │       ├── __init__.py
│   │   │       ├── test_management_commands.py
│   │   ├── utils
│   │   │   ├── __init__.py
│   │   │   ├── management_commands.py

And a sample output of the coverage report:

coverage run --source apps/ -m py.test apps/
coverage report

Name                                        Stmts   Miss  Cover
---------------------------------------------------------------
apps/api/views/tickets.py                      42     18    57%
apps/support/utils/management_commands.py     135    100    26%    

Looking at the html report I can see many statements executed by the tests are not considered covered, even though they should be. I believe this coverage data to be incomplete, it seems to only be considering imports, definitions, and docstrings as covered.

Unable to determine why the coverage appeared incorrect, I tried running a single test module, with positive results:

coverage run --source apps/support/utils/management_commands.py -m py.test apps/support/tests/utils/test_management_commands.py
coverage report

Name                                        Stmts   Miss  Cover
---------------------------------------------------------------
apps/support/utils/management_commands.py     135     68    50%

This is more accurate, the HTML report shows the statement I have tests for are indicated as covered this time. Unable to figure out why running a single test module yields accurate results, I modified the directory structure by moving the tests under a single parent folder.

.
├── apps
│   ├── api
│   │   ├── __init__.py
│   │   └── views
│   │       ├── __init__.py
│   │       ├── tickets.py
│   ├── support
│   │   ├── __init__.py
│   │   ├── utils
│   │   │   ├── __init__.py
│   │   │   ├── management_commands.py
├── tests
│   │   ├── __init__.py
│   ├── api
│   │   ├── __init__.py
│   │   └── views
│   │       ├── __init__.py
│   │       └── test_tickets.py
│   ├── support
│   │   ├── __init__.py
│   │   ├── utils
│   │   │   ├── __init__.py
│   │   │   ├── test_management_commands.py

Re-running coverage with this directory structure yields more accurate results:

coverage run --source apps/ -m py.test tests/
coverage report

Name                                        Stmts   Miss  Cover
---------------------------------------------------------------
apps/api/views/tickets.py                      42      0   100%
apps/support/utils/management_commands.py     135     68    50%

Can anyone explain why running coverage with py.test was yielding in complete coverage under the original directory structure? Was the directory structure actually the problem or am I missing something else here?

Additional info:

# pytest.ini
[pytest]
addopts = --nomigrations
markers =
    slowtest: mark a test as being slow
    integration: mark a test as being an integration test

INSTALLED_APPS += ('django_coverage', )
TEST_DISCOVER_PATTERN = 'test_*'
COVERAGE_MODULE_EXCLUDES = [
    'settings',
    'urls$',
    'locale$',
    'tests$',
    'django',
    'migrations',
    'compressor',
    'templates?$',
    'fixtures$',
    'static$',
]
ROOT_PATH = os.path.abspath('%s/' % os.path.dirname(__file__))

The .coveragerc

[run]
source = apps
omit =
     apps/*/templates?/*
     apps/*/migrations/*
     apps/*/factories/*
     apps/*/tests/*
[html]
directory = coverage

Module versions (some may be unrelated):

pytest==2.9.0
pytest-cov==2.2.1
pytest-django==2.9.1
django-coverage==1.2.4
coverage==4.0.3

Answer 1

I see I'm a little late to the party (3-year-old question with no accepted answer yet), but since I've just had this same question with a seemingly obvious answer: coverage will only report on code that is actually run. So if your tests don't call a bit of code and it doesn't get run during normal loading of the application, coverage will not show a report for that code. Code that doesn't run does not cause bugs :)