unittest is not able to discover / run tests

Question

There are some related questions, but none apply.

This is my directory tree:

» tree abc_backend
abc_backend/
├── backend_main.py
├── FundDatabase.db
├── healthcheck.py
├── __init__.py
├── init.py
├── portfolio.py
├── private.py
├── __pycache__
├── questionnaire.py
├── recurring.py
├── registration.py
├── tests
│   ├── config.py
│   ├── __init__.py
│   ├── __pycache__
│   ├── test_backend.py
│   ├── test_healthcheck.py
│   └── test_private.py
├── trading.py
├── Users.db
├── VERSION
└── visualisation.py

unittest is not able to find anything:

top » python -m unittest abc_backend

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

Not even from within abc_backend:

abc_backend » python -m unittest tests

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

What I have already verified:

• my test methods are properly named (test_whatever)
• my testcases extend unittest.TestCase
• the abc_backend and the abc_backend/tests directories have an (empty) __init__.py
• all test modules are importable (see below)
• unittest discover finds the tests, but has problems with relative imports (see below)
• nose is able to discover and run the tests, no problems

I would like to understand:

• why do I need to pass discover to unittest to force it to discover the tests? What does unittest do without the discover sub-command? (I thought unittest does test discovery by default). According to the documentation:

python -m unittest is the equivalent of python -m unittest discover

• once the tests are discovered (by forcing the discover sub-command), why do I have import issues?

Test modules are importable

» python
Python 3.4.3 (default, Oct 14 2015, 20:28:29) 
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import abc_backend.tests
>>> import abc_backend.tests.test_private
>>> import abc_backend.tests.test_healthcheck
>>> import abc_backend.tests.test_backend

unittest discover has problems with relative imports

If I run it from the top dir:

top » python -m unittest discover abc_backend
======================================================================
ERROR: tests.test_private (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.4/unittest/case.py", line 58, in testPartExecutor
    yield
  File "/usr/lib/python3.4/unittest/case.py", line 577, in run
    testMethod()
  File "/usr/lib/python3.4/unittest/loader.py", line 32, in testFailure
    raise exception
ImportError: Failed to import test module: tests.test_private
Traceback (most recent call last):
  File "/usr/lib/python3.4/unittest/loader.py", line 312, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.4/unittest/loader.py", line 290, in _get_module_from_name
    __import__(name)
  File "/foo/bar/abc_backend/tests/test_private.py", line 6, in <module>
    from .. import init
ValueError: attempted relative import beyond top-level package

If I run it from within abc_backend:

abc_backend » python -m unittest discover tests

======================================================================
ERROR: test_private (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.4/unittest/case.py", line 58, in testPartExecutor
    yield
  File "/usr/lib/python3.4/unittest/case.py", line 577, in run
    testMethod()
  File "/usr/lib/python3.4/unittest/loader.py", line 32, in testFailure
    raise exception
ImportError: Failed to import test module: test_private
Traceback (most recent call last):
  File "/usr/lib/python3.4/unittest/loader.py", line 312, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.4/unittest/loader.py", line 290, in _get_module_from_name
    __import__(name)
  File "/foo/bar/abc_backend/tests/test_private.py", line 6, in <module>
    from .. import init
SystemError: Parent module '' not loaded, cannot perform relative import

Answer 1

I reproduced all the problems with CPython 3.5, so my answer should be relevant to both 3.4 and 3.5.

Relative import issues

The reason why there are issues with relative imports is that due to specifics of invocations you really do not import abc_backend package.

First, let’s take a look at

top» python3 -m unittest discover abc_backend

When you run tests from top that way, abc_backend is just not imported. That is because /home/user/top/abc_backend is added to sys.path instead of /home/user/top. To solve this problem, do

top» python3 -m unittest discover abc_backend -t .

Now, about the in-abc_backend invocation. When you do

abc_backend» python3 -m unittest discover tests

abc_backend is not importable, as /home/user/top/abc_backend/tests dir does not contain abc_backend package. This too can be solved with

abc_backend» python3 -m unittest discover tests -t ../

that will correctly put /home/user/top dir (pun intended) into sys.path.

The -t (or --top-level-directory) option sets top level directory of project and defaults to start directory (which is . by default). So, what is in sys.path is important, as that affects imports, which affect test loading, as discovery loads tests using import machinery.

Difference between -m unittest and -m unittest discover

When you do

top» python3 -m unittest abc_backend

in reality you are running unittest/__main__.py file. There main(module=None) is invoked, and eventually you get to loadTestsFromModule that does

tests = []
for name in dir(module):
    obj = getattr(module, name)
    if isinstance(obj, type) and issubclass(obj, case.TestCase):
        tests.append(self.loadTestsFromTestCase(obj))

As abc_backend/__init__.py does not contain any test cases, isinstance(obj, type) and issubclass(obj, case.TestCase) returns False for all module members (so tests is empty).

To make this particular way of invocation work, you’ll have to do what people usually did in pre-discover times (aside from non-stdlib frameworks): manually import cases from test modules (or maybe construct test suite according to load_tests protocol).

So, how

top» python3 -m unittest discover abc_backend

differs?

Basically, differences may be expressed as following conditional:

if len(argv) > 1 and argv[1].lower() == 'discover':
    # -m unittest discover
    loader.discover(...)
else:
    # -m unittest
    loader.loadTestsFromNames(...)

When argv is ['python3 -m unittest', 'discover', 'abc_backend'], actual discovery mechanism is used. When argv is ['python3 -m unittest', 'abc_backend'], loadTestsFromNames is used, which calls loadTestsFromModule at some point, and no tests are found. That’s the way things are in unittest/main.py.