Avoid package name collision in Python Unittest

Question

Here is my project layout:

project
+-- package_1
|   +-- __init__.py
|   +-- module_1.py tests
+-- package_2
|   +-- __init__.py
|   +-- module_2.py tests
+-- tests
    +-- package_1
    |   +-- __init__.py
    |   +-- test_module_1.py
    +-- package_2
        +-- __init__.py
        +-- test_module_2.py

test_module_1.py starts with:

import package_1.module_1

test_module_2.py starts with:

import package_2.module_2

Running python -m unittest discover tests from the project directory gives errors:

EE
======================================================================
ERROR: package_1.test_module_1 (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: package_1.test_module_1
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/loader.py", line 434, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/loader.py", line 375, in _get_module_from_name
    __import__(name)
  File "/Users/maggyero/project/tests/package_1/test_module_1.py", line 1, in <module>
    import package_1.module_1
ModuleNotFoundError: No module named 'package_1.module_1'


======================================================================
ERROR: package_2.test_module_2 (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: package_2.test_module_2
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/loader.py", line 434, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/loader.py", line 375, in _get_module_from_name
    __import__(name)
  File "/Users/maggyero/project/tests/package_2/test_module_2.py", line 1, in <module>
    import package_2.module_2
ModuleNotFoundError: No module named 'package_2.module_2'


----------------------------------------------------------------------
Ran 2 tests in 0.000s

FAILED (errors=2)

Adding import sys; print(sys.modules['package_1']) at the beginning of test_module_1.py and import sys; print(sys.modules['package_2']) at the beginning of test_module_2.py to see what is in the sys.modules cache shows that package_1 and package_2 from the tests directory have been already imported during the test discovery:

<module 'package_1' from '/Users/maggyero/project/tests/package_1/__init__.py'>
<module 'package_2' from '/Users/maggyero/project/tests/package_2/__init__.py'>

Importing a previously imported package reuse the same cached package from sys.modules, even if sys.path have since been updated. So when import package_1.module_1 and import package_2.module_2 are executed, first package_1 and package_2 from the tests directory (which contain test_module_1 and test_module_2) are reimported instead of package_1 and package_2 from the project directory (which contain module_1 and module_2), then module_1 and module_2 are imported, raising a ModuleNotFoundError.

Is there a workaround to avoid that the packages from the tests directory shadow those of the project directory, besides renaming?

Update (post answer)

An alternative solution to Laurent Laporte's below (his avoids having 'package_1' and 'package_2' already in sys.modules when executing import package_1.module_1 and import package_2.module_2, by having 'tests.package_1' and 'tests.package_2' instead, thanks to a change of the top-level directory) is to update sys.path and reload the packages in test_module_1.py:

import importlib
import pathlib
import sys
sys.path.insert(0, pathlib.Path(__file__).resolve().parents[2])

import package_1
importlib.reload(package_1)
import package_1.module_1

and test_module_2.py:

import importlib
import pathlib
import sys
sys.path.insert(0, pathlib.Path(__file__).resolve().parents[2])

import package_2
importlib.reload(package_2)
import package_2.module_2

The only advantage of this solution is that the tests directory does not need to be a regular package (that is having an __init__.py file). So there won't be any advantage when Unittest allows recursive namespace package discovery (for the moment the ticket is still open: https://bugs.python.org/issue23882).

Laurent Laporte's solution should be preferred, as package qualification is better to distinguish packages with the same names than package reloading. Another good solution is package renaming (for instance renaming package_1 and package_2 from the tests directory to test_package_1 and test_package_2).

Answer 1

You can solve your problem by using the flag -t, --top-level-directory directory to set the top level directory of your project (defaults to start directory)

For instance:

python -m unittest discover tests -t .

But, in order for discover to import your test modules you need to turn your tests directory into a package by inserting a __init__.py in it.

See the documentation about tests discovery.

NOTES:

• I encounter the same problem with PyTest. See my study on GitHub.

• In other open source projects, there is only one root package (only package_1 for instance), and there is no tests/package_1 directory, only tests with all the test_*.py modules (and possibly with sub-packages). So, the problem doesn't appear.