How to include examples or test programs in a package?

Question

The Python Cookbook suggests the following tree structure for a "typical library package":

projectname/
    README.txt
    Doc/
        documentation.txt
    projectname/
        __init__.py
        foo.py
        bar.py
        utils/
            __init__.py
            spam.py
            grok.py
    examples/
        helloworld.py

You 'll notice that the examples/ are not part of the actual package, which resides under projectname/projectname/ (that's where you 'll find the top-level __init__.py of the package).

Well, examples/helloworld.py obviously needs to import the projectname package.

I am aware that there are at least 2-3 relevant questions in StackOverflow. I do not believe that this is a duplicate because the other questions either involve intra-package imports or the general case of importing one python module from another when they do not reside in the same directory. I am specifically asking for the suggested way to do this when packaging a library.

Is there a way to achieve this without modifying the path? If modifying the path is the only way, is there a way for this to be done in an elegant manner?

Let me elaborate on that last point. In Repository Structure and Python by Kenneth Reitz, a similar structure appears, with tests/ instead of examples/. This is exactly the same problem. He suggests using "a simple (but explicit) path modification to resolve the package properly." OK, but this is the actual code:

import os
import sys
sys.path.insert(0, os.path.abspath('..'))

I really don't like the .. part. I would hope for a more general solution, hopefully one that would work from whichever directory I would choose to run the example (or the test).

Answer 1

While the folder tree looks alright, I believe that implicitly assuming that modules are available for import is wrong from both a developer's perspective and certainly from a user's perspective.

It's true that you can technically use sys.path.insert(0, os.path.abspath('..')) to add any path for python to allow imports from, but that means you, the developer, have to make sure that the added path is always in the right location.

It is common for users to install packages to use them. There's a clear workflow for developers:

1. Have pip and virtualenv installed (and even better, virtualenvwrapper)
2. Install the package in editable mode using pip's -e flag which means that any changes you make to the code will directly effect execution. You won't have to reinstall everytime you make changes to your code.
3. Since your code is always installed (specifically, under site-packages as a user but in editable mode when you develop), you can always import your package using its explicit name from any example or test.

A common workflow:

$ pip install virtualenv
...
$ virtualenv distro
New python executable in /home/nir0s/work/distro/bin/python3
Also creating executable in /home/nir0s/work/distro/bin/python
Installing setuptools, pip, wheel...done.

$ source distro/bin/activate

# install in editable mode
$ pip install -e ~/repos/nir0s/distro/
Obtaining file:///home/nir0s/repos/nir0s/distro
Installing collected packages: distro
  Running setup.py develop for distro
Successfully installed distro

(distro) $ pip freeze
appdirs==1.4.3
-e [email protected]:nir0s/distro@e8a182f9d1dbe6391f25...#egg=distro
packaging==16.8
pyparsing==2.2.0
six==1.10.0

The package, being installed in editable mode, means that there's an egg-link file pointing to the directory of the package:

$ cat distro/lib/python3.6/site-packages/distro.egg-link 
/home/nir0s/repos/nir0s/distro

Users will do the same thing without dealing with editable mode. Simply creating virtual environments and install packages in them. Then, when they're done working, they'll delete those virtual environments. Easy peasy.