Building a package with generated Python files

Question

Problem statement

When building a Python package I want the build tool to automatically execute the steps to generate the necessary Python files and include them in the package.

Here are some details about the project:

• the project repository contains only the hand-written Python and YAML files
• to have a fully functional package the YAML files must be compiled into Python scripts
• once the Python files are generated from YAMLs, the program needed to compile them is no longer necessary (build dependency).
• the hand-written and generated Python files are then packaged together.

The package would then be uploaded to PyPI.

I want to achieve the following:

• When the user installs the package from PyPI, all necessary files required for the package to function are included and it is not necessary to perform any compile steps
• When the user checks-out the repository and builds the package with python -m build . --wheel, the YAML files are automatically compiled into Python and included in the package. Compiler is required.
• When the user checks-out the repository and installs the package from source, the YAML files are automatically compiled into Python and installed. Compiler is required.
• (nice to have) When the user checks-out the repository and installs in editable mode, the YAML files are compiled into Python. The user is free to make modifications to both generated and hand-written Python files. Compiler is required.

I have a repository with the following layout:

├── <project>
│   └── <project>
│       ├── __init__.py
│       ├── hand_written.py
│       └── specs
│           └── file.ksc (YAML file)
└── pyproject.toml

And the functional package should look something like this

├── <project>
│   └── <project>
│       ├── __init__.py
│       ├── hand_written.py
│       └── generated
│           └── file.py
├── pyproject.toml
└── <other package metadata>

How can I achieve those goals?

What I have so far

As I am very fresh to Python packaging, I have been struggling to understand the relations between the pyproject.toml, setup.cfg and setup.py and how I can use them to achieve the goals I have outlined above. So far I have a pyproject.toml with the following content:

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
name = "<package>"
version = "xyz"
description = "<description>"
authors = [ <authors> ]
dependencies = [
    "kaitaistruct",
]

From reading the setuptools documentation, I understand that there are the build commands, such as:

• build_py -- simply copies Python files into the package (no compiling; works differently in editable mode)
• build_ext -- builds C/C++ modules (not relevant here?)

I suppose adding the compile steps for the YAML files will involve writing a setup.py file and overwriting a command, but I don't know if this is the right approach, whether it will even work, or if there are better methods, such as using a different build backend.

Alternative approaches

A possible alternative approach would be to manually compile the YAML files prior to starting the installation or build of the package.

Answer 1

I think I got an answer to my own question. What I basically needed was setuptools sub-commands. Sub-commands allow the build process to be extended with custom steps without having to over-write an existing command.

First, I needed to create setup.py file with a class that implements the Command protocol.

from setuptools import Command

class BuildKsy(Command):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.build_lib = None
        self.editable_mode = False

    def initialize_options(self):
        """Initialize command state to defaults"""
        ...

    def finalize_options(self):
        """
        Populate the command state. This is where I traverse the directory
        tree to search for the *.ksc files to compile them later.
        The self.set_undefined_options is used to inherit the `build_lib`
        attribute from the `build_py` command.
        """
        self.set_undefined_options("build_py", ("build_lib", "build_lib"))
        ...

    def run(self):
        """
        Perform actions with side-effects, such as invoking a ksc to python compiler. 
        The directory to which outputs are written depends on `editable_mode` attribute.
        When editable_mode == False, the outputs are written to directory pointed by build_lib.
        When editable_mode == True, the outputs are written in-place, 
        i.e. into the directory containing the sources.
        The `run` method is not executed during sdist builds.
        """
        ...

    def get_output_mapping(self):
        """
        Return dict mapping output file paths to input file paths
        Example:
            dict = { "build/lib/output.py": "project/specs/file.ksc" }
        """
        ...

    def get_outputs(self):
        """Return list containing paths to output files (generated *.py files in my case)"""
        ...

    def get_source_files(self):
        """Returns list containing paths to input files (*.ksc YAMLs in my case)"""

Finally I needed to register the custom build sub-command.

import setuptools.command.build
setuptools.command.build.build.sub_commands.append(("build_ksy", None))

The sub_commands.append() takes a tuple containing the command name and a second parameter that can either be None or a method.

The documentation is a bit lacking in this area, but looking at the source code I figured out what the second parameter is. If the second parameter is a method, then it will be executed. The method must return a boolean value. If the method returns True, the sub-command will be executed in the build process. If the second parameter is None, then the sub-command will also be executed in the build process. Otherwise the sub-command will not be executed. Here is the piece of code in setuptools that handles it.

Lastly, it is necessary to call setuptools.setup(). The cmdclass parameter must map the command name from the sub_commands.append tuple parameter to the Python class implementing the command.

setuptools.setup(cmdclass={"build_ksy": BuildKsy})

After going through the above steps it is possible to simply use

python -m build . --wheel
python -m build . --sdist

to build wheels and source distributions respectively. The wheel will contain the fully processed distribution that does not require the user to perform the ksc compile step. The source distribution however will still require the user to execute the ksc compile step and requires the compiler to be installed.

It is also possible to have an editable installation by calling

pip install -e <package-dir>