Building Python Package with source in different directory

Question

When building a python package where the source tree looks like this:

src -\
     +- module -\
           <stuff>
     +- setup.py

is pretty clear.

Is it possible to build a package where module source doesn't reside in the same location as the setup.py? For more specific use case the code for the module is either partially or full autogenerated in a location other then src

E.g.

src -\
     +- setup.py
generated -\
           module -\
                   <module code>

Answer 1

You can control the directory where packages reside by using the package_dir argument to setup(...)

and while it does appear to build a proper source distribution when package_dir is a relative path starting with .., it appears that pip will refuse to install it -- I'd suggest instead nesting your generated code inside that src directory instead and then using package_dir to select that.

Here's an example which moves all modules inside a generated subdir:

setup(
    name='mypkg',
    package_dir={'': 'generated'},
    packages=find_packages('generated'),
)

Using a setup like:

$ tree .
.
├── generated
│   ├── mod1
│   │   └── __init__.py
│   └── mod2
│       └── __init__.py
└── setup.py

This would make the following succeed after install: import mod1; import mod2

If you wanted to make those modules available under a different prefix, you would do:

setup(
    name='mypkg',
    package_dir={'hello': 'generated'},
    packages=[f'hello.{mod}' for mod in find_packages('generated')],
)

This would make import hello.mod1; import hello.mod2 succeed after installation

Answer 2

You can use relative paths in package lookup configuration. Examples:

all distributed sources are in generated

from setuptools import setup, find_packages


setup(
    ...
    package_dir={'': '../generated'},
    packages=find_packages(where='../generated'),
)

selected packages should be included from generated

In this example, only packages spam and eggs from generated will be included:

import pathlib
from setuptools import setup, find_packages


setup(
    name='so',
    package_dir={'spam': '../generated/spam', 'eggs': '../generated/eggs'},
    packages=find_packages(where='../generated'),  # or just ['spam', 'eggs']
)

Or implement a dynamic lookup like e.g.

package_dir={p.name: p.resolve() for p in pathlib.Path('..', 'generated').iterdir()}

better implementation by resolving all paths relative to the setup.py file

Resolving all paths relative to the setup.py script allows you to run the script from any other directory than src, e.g. you can run python src/setup.py bdist_wheel etc. You may or may not need it, depending on your use case. Nevertheless, the recipe is as usual: resolve all paths to __file__, e.g.

import pathlib
from setuptools import setup, find_packages


src_base = pathlib.Path(__file__, '..', '..', 'generated').resolve()

setup(
    ...
    package_dir={'': str(src_base)},
    packages=find_packages(where=src_base),
)