How can I make pip install package data (a config file)?

Question

I have a package called clana (Github, PyPI) with the following structure:

.
├── clana
│   ├── cli.py
│   ├── config.yaml
│   ├── __init__.py
│   ├── utils.py
│   └── visualize_predictions.py
├── docs/
├── setup.cfg
├── setup.py
├── tests/
└── tox.ini

The setup.py looks like this:

from setuptools import find_packages
from setuptools import setup

requires_tests = [...]

install_requires = [...]


config = {
    "name": "clana",
    "version": "0.3.6",
    "author": "Martin Thoma",
    "author_email": "[email protected]",
    "maintainer": "Martin Thoma",
    "maintainer_email": "[email protected]",
    "packages": find_packages(),
    "entry_points": {"console_scripts": ["clana=clana.cli:entry_point"]},
    "install_requires": install_requires,
    "tests_require": requires_tests,
    "package_data": {"clana": ["clana/config.yaml"]},
    "include_package_data": True,
    "zip_safe": False,
}

setup(**config)

How to check that it didn't work

Quick

python3 setup.py sdist
open dist/clana-0.3.8.tar.gz  # config.yaml is not in this file

The real check

I thought this would make sure that the config.yaml is in the same directory as the cli.py when the package is installed. But when I try this:

virtualenv venv
source venv/bin/activate
pip install clana
cd venv/lib/python3.6/site-packages/clana
ls

I get:

cli.py  __init__.py  __pycache__  utils.py  visualize_predictions.py

The way I upload it to PyPI:

python3 setup.py sdist bdist_wheel && twine upload dist/*

So the config.yaml is missing. How can I make sure it is there?

Answer 1

You can add a file name MANIFEST.in next to setup.py with a list of the file you want to add, wildcard allowed (ex: include *.yaml or include clana/config.yaml) then the option include_package_data=True will activate the manifest file

Answer 2

In short: add config.yaml to MANIFEST.in, and set include_package_data. One without the other is not enough.

Basically it goes like this:

1. MANIFEST.in adds files to sdist (source distribution).
2. include_package_data adds these same files to bdist (built distribution), i.e. it extends the effect of MANIFEST.in to bdist.
3. exclude_package_data prevents files in sdist to be added to bdist, i.e. it filters the effect of include_package_data.
4. package_data adds files to bdist, i.e. it adds build artifacts (typically the products of custom build steps) to your bdist and has of course no effect on sdist.

So in your case, the file config.yaml is not installed, because it is not added to your bdist (built distribution). There are 2 ways to fix this depending on where the file comes from:

• either the file is a build artifact (typically it is somehow created during the ./setup.py build phase), then you need to add it to package_data ;

• or the file is part of your source (typically it is in your source code repository), then you need to add it to MANIFEST.in, set include_package_data, and leave it out of exclude_package_data (this seems to be your case here).

See:

• https://stackoverflow.com/a/54953494/11138259
• https://setuptools.readthedocs.io/en/latest/setuptools.html#including-data-files