How to properly package a project with Setuptools in Python?

Question

Here is my directory tree:

prediction_model
├─ prediction_model
|  ├─ __init__.py
|  ├─ data
|  |  ├─ SAAA.csv
|  |  └─ VDFF.csv
|  ├─ models.py
|  ├─ preprocess.py
|  ├─ README.md
|  └─ tests
└─ setup.py

Here is my ‘setup.py’:

from setuptools import find_packages, setup

setup(
    name='prediction_model',
    version='0.7',
    url='https://project.org/',
    author='JL',
    author_email='[email protected]',
    packages=find_packages(),
    scripts=['models.py', 'preprocess.py']
)

Here is my ‘__init__.py’:

from prediction_model import models
from prediction_model import preprocess

‘models.py’ has a function main and ‘preprocess.py’ has a function run that I want to use.

I install the project using:

python -m pip install --user .

Then I run the following code in the Python interpreter but it raises an exception AttributeError:

>>> import prediction_model
>>> prediction_model.src.main()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'prediction_model' has no attribute 'src'
>>> prediction_model.src.run()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'prediction_model' has no attribute 'src'

>>> import prediction_model
>>> prediction_model.main()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'prediction_model' has no attribute 'main'
>>> prediction_model.run()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'prediction_model' has no attribute 'run'

What am I doing wrong?

Environment: Python 3.7, MacOS.

Answer 1

Set up

The best practice is to use the src layout (cf. https://blog.ionelmc.ro/2014/05/25/python-packaging/), so I suggest that you organize your directories like this (since you should not package the ‘tests’ directory and ‘README.rst’ file for built distribution, they are left outside the ‘src’ directory):

project
├─ src
|  └─ package
|     ├─ __init__.py
|     ├─ __main__.py
|     ├─ data.tsv
|     └─ module.py
├─ tests
|  └─ test_module.py
├─ MANIFEST.in
├─ README.rst
└─ setup.py

with the following ‘setup.py’ file contents:

import setuptools

setuptools.setup(
    name="project name",
    version="project version",
    url="project URI",
    author="your name",
    author_email="your email",
    package_dir={"": "src"},
    packages=setuptools.find_namespace_packages(where="src"),
    include_package_data=True
)

and the following ‘MANIFEST.in’ file contents:

graft src
graft tests
global-exclude *.py[cod]

Remark. — If you don't want a top-level ‘package’ directory in your ‘src’ directory while keeping the ‘src’ directory (like before your post edit):

project
├─ src
|  ├─ __init__.py
|  ├─ __main__.py
|  ├─ data.tsv
|  └─ module.py
├─ tests
|  └─ test_module.py
├─ MANIFEST.in
├─ README.rst
└─ setup.py

then map "package" to "src" in the package_dir argument and list "package" explicitly in the packages argument of the setuptools.setup function in the setup.py file (cf. https://docs.python.org/3/distutils/examples.html#pure-python-distribution-by-package):

import setuptools

setuptools.setup(
    name="project name",
    version="project version",
    url="project URI",
    author="your name",
    author_email="your email",
    package_dir={"package": "src"},
    packages=["package"],
    include_package_data=True
)

Package

Now you can package the project into a source distribution with this command:

python setup.py sdist

or a built distribution with this command (this is an extension command to setuptools requiring that wheel be installed in your Python environment—so run pip install wheel before):

python setup.py bdist_wheel

or both with this command:

python setup.py sdist bdist_wheel

which create the following files in a new dist directory:

• ‘{project name}-{project version}.tar.gz’ for a source distribution;
• ‘{project name}-{project version}-{compatibility tags}.whl’ for a built distribution.

Install

Now you can install the project from the source distribution with this command:

pip install dist/{project name}-{project version}.tar.gz

or from the built distribution with this command:

pip install dist/{project name}-{project version}-{compatibility tags}.whl

or from the project tree with this command (no need to package the project for this one):

pip install .

Run

Eventually you can import your project like this:

import package.module

package.module.function()