Does it make sense to install my Python unit tests in site-packages?

Question

I'm developing my first Python distribution package. My learning curve on Python packaging seems to be leveling off a bit, but I'm still wrestling with a few open questions. One is whether I should cause my unit tests to be installed alongside my code.

I understand it's important to include tests in a source distribution. What I'm wondering is whether I should actually configure them to be installed?

I've seen at least one popular package that appears to do this on purpose (PyHamcrest), and at least one other that appears to do it by accident (behave).

So my (multi-part) question is this:

• Does it ever make sense to install my package unit tests alongside my package code?

• If so, what is the use case? Who would use them and for what? That is, who would use them that wouldn't be perfectly happy to download the source distribution and run python setup.py test instead?

• And how would they use installed unit tests? like import test; test.run() or something like that?

Answer 1

In my opinion the right answer is NO but you will find quite a few distributions that install the tests. Tests shouldn't be installed but they should be included in the source distribution. In my opinion in an ideal world testing installed packages should be a task performed by the package manager (pip) and the site-packages directory shouldn't be polluted with test sources.

I've recently researched this topic and gathered information from various sources and found several different ways to structure the directory/package hierarchy of a distribution that contains both library sources and tests. Most of those structures seem to be obsolete and they were invented as attempts to work around the incomplete feature-sets of older distribution systems at the time. Unfortunately a lot of online sources (older blogposts/documentation) are still advertising the outdated methods so it's very easy to find an outdated distribution how-to/tutorial with online search.

Let's assume you have a library called "my_lib" and you want to structure the sources of your distribution. I will show two popular and seemingly outdated ways to structure your distribution and a third way I've found to be the most versatile. The third method may also be outdated but that's the best one I know a the time of posting this answer. ;-)

Method #1

Distributions that (intentionally or unintentionally) install tests usually use this method.

hierarchy

+- my_lib
|  +- __init__.py
|  +- source1.py
|  +- source2.py
|  +- tests
|     +- __init__.py
|     +- test_1.py
|     +- test_2.py
+- setup.py

Method #2

Tests aren't installed but they should be included in the source distribution through the MANIFEST.in file.

hierarchy

+- my_lib
|  +- __init__.py
|  +- source1.py
|  +- source2.py
+- tests
|  +- __init__.py
|  +- test_1.py
|  +- test_2.py
+- setup.py

Method #3 (I prefer this one.)

This is pretty much similar to Method #2 with a little twist (the src dir).

hierarchy

+- src
|  +- my_lib
|     +- __init__.py
|     +- source1.py
|     +- source2.py
+- tests
|  +- __init__.py
|  +- test_1.py
|  +- test_2.py
+- setup.py

setup() call in setup.py

from setuptools import setup, find_packages

setup(
    ...
    packages=find_packages('src'),
    package_dir={'': 'src'},
    ...
)

MANIFEST.in

recursive-include tests *.py

Tests won't be installed but they will be included in the source distribution through our MANIFEST.in.

In case of method #3 you have an src directory that usually contains only a single package that is the root of your lib. Putting the my_lib package into an src directory (directory and not a package so you don't need an src/__init__.py) has the following benefits:

• When you execute setup.py the directory that contains setup.py is implicitly added to the python path. This means that in your setup.py you can accidentally and improperly import stuff from your library if it's package is in the same directory as setup.py. By putting the my_lib package into src we can avoid this problem.

• You can easily use your distributed test sources to test both the distributed library sources and also the installed library:

  • When you run the tests with setup.py test the package_dir={'': 'src'} part of your setup() call guarantees that your tests will see your my_lib library package that you keep in src/my_lib.
  • You can also run tests without setup.py. In the simplest case you can do that with the python -m unittest command. In this case the src dir won't be part of the python path so you can use this method to test the installed version of the library instead of the sources in src.