Is pytest supposed to collect tests from dependency modules in a virtual environtment?

Question

I am attempting to setup a project on another laptop than my typical development machine. This project has several pytest-based tests that I have written over the lifetime of the project. When I run

$ pytest -k tests/my_test.py

I get a list of errors from sqlalchemy tests like the following:

_ ERROR collecting env/lib64/python3.5/site-packages/sqlalchemy/testing/suite/test_update_delete.py _
env/lib/python3.5/site-packages/py/_path/local.py:662: in pyimport
    __import__(modname)
env/lib/python3.5/site-packages/sqlalchemy/testing/suite/__init__.py:2: in <module>
    from sqlalchemy.testing.suite.test_cte import *
<frozen importlib._bootstrap>:968: in _find_and_load
    ???
<frozen importlib._bootstrap>:957: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:664: in _load_unlocked
    ???
<frozen importlib._bootstrap>:634: in _load_backward_compatible
    ???
env/lib/python3.5/site-packages/_pytest/assertion/rewrite.py:211: in load_module
    py.builtin.exec_(co, mod.__dict__)
env/lib/python3.5/site-packages/sqlalchemy/testing/suite/test_cte.py:11: in <module>
    class CTETest(fixtures.TablesTest):
env/lib/python3.5/site-packages/sqlalchemy/testing/suite/test_cte.py:99: in CTETest
    @testing.requires.ctes_with_update_delete
E   AttributeError: 'NoneType' object has no attribute 'ctes_with_update_delete'

Why is pytest collecting tests from a dependency? Is it supposed to do this? If not, how can I fix it?

The laptop where I am setting up this project is running Ubuntu 18.04. I created a virtualenv with Python 3.5.6 and ran pip install -r requirements in the virtualenv without any errors, including sqlalchemy and pymssql.

Answer 1

When pytest is invoked, it scans all the child dirs in the project root dir, looking for tests; when you place the project-specific virtual env into project root dir, it will be also scanned without making an exception. This can lead to unwanted tests being included in the test run (for example, when you have dependencies like numpy or pandas that include tests in the distribution).

To circumvent this, pytest offers two configuration options:

1. norecursedirs - holds directories that will be excluded from scanning. Use this option when you are looking for the pattern "include all, exclude selected". See also wim's answer for the explanation and usage example. By default, norecursedirs is set to .*', 'build', 'dist', 'CVS', '_darcs', '{arch}', '*.egg', so beware that when you override this option, the defaults are gone and you have to add them back.

2. testpaths - holds directories that should only be considered for the scan, so this is basically the opposite to what norecursedirs is doing. Use this option when looking for the pattern "exclude all, include selected". This option also adds some minor or more significant speedup to the test discovery, depending on what you keep in the project root - most of the sudbirectories won't be traversed at all and the tests run starts sooner.

   Usage: either place the options in the pytest.ini/setup.cfg/tox.ini:

   [tool:pytest]
   testpaths = tests othertests doc
   

   or pass them via --override-ini from command line.

   pytest -o "testpaths=tests othertests doc" ...
   

Moving the env dir away from the project sources is a matter of choice and depends on what dev environment you have. direnv, pipenv etc. may have different views on where the envs should reside, but personally I don't see any problems in keeping the envs in the project root.

Answer 2

To answer the question in the title, no.

This may be a symptom of creating your venv and installing packages in a directory that is not ignored. Keep them somewhere ignored, like ~/.venvs/ or whatever.

Alternatively, specifically --ignore=env to exclude the site, or otherwise customize your test discovery in config file:

# content of pytest.ini
[pytest]
norecursedirs = env