How to Register Multiple Pluggy Plugins with Setuptools

Question

Problem

I can successfully register a single plugin in pluggy using load_setuptools_entrypoints, but I can only register one. If two different plugins try to register themselves, the last one registered will be the only one that runs.

I think this is not how pluggy is intended to work, and that I am making a configuration mistake, but I don't know what to do differently. I think pluggy is supposed to allow multiple plugins to register and run in serial when that hook is called.

Project Structure

.
├── pluggable
│   ├── pluggable.py
│   └── pyproject.toml
├── plugin_a
│   ├── a.py
│   └── pyproject.toml
└── plugin_b
    ├── b.py
    └── pyproject.toml

with:

# pluggable/pluggable.py
import pluggy

NAME = "pluggable"
impl = pluggy.HookimplMarker(NAME)


@pluggy.HookspecMarker(NAME)
def run_plugin():
    pass


def main():
    m = pluggy.PluginManager(NAME)
    m.load_setuptools_entrypoints(NAME)
    m.hook.run_plugin()


if __name__ == "__main__":
    main()

and

# pluggable/pyproject.toml
[project]
name = "pluggable"
version = "1.0.0"
dependencies = ["pluggy==1.3.0"]

Both plugin_a/a.py and plugin_b/b.py have the same contents:

# plugin_a/a.py
import pluggy

from pluggable import impl


@impl
def run_plugin():
    print(f"run from {__name__}")


if __name__ == "__main__":
    run_plugin()

plugin_a/pyproject.toml and plugin_b/pyproject.toml register their respective modules as pluggy plugins:

# plugin_a/pyproject.toml
[project]
name = "plugin_a"
version = "1.0.0"
dependencies = ["pluggy==1.3.0", "pluggable"]

[project.entry-points.pluggable]
run_plugin = "a"

plugin_b/pyproject.toml is the same, except that run_plugin = "b".

Demo run

$ python -m venv venv
...
$ venv/bin/pip install -e pluggable -e plugin_a 
...
Successfully installed pluggable-1.0.0 plugin-a-1.0.0
$ venv/bin/python pluggable/pluggable.py
run from a
$ venv/bin/pip install -e plugin_b
...
Successfully installed plugin-b-1.0.0
$ venv/bin/python pluggable/pluggable.py
run from b

Expected outcome

What I'd like to see at the end is

run from b
run from a

Further analysis

From reading the load_setuptools_entrypoints code it's pretty clear what's happening. As soon as one plugin is registered to a name, PluginManager.get_plugin(name) returns that value, so other plugins will not be registered at the same name.

But what I am hoping to get help to understand is what is the correct way to configure a plugin system with pluggy so that two different python packages can register themselves with the same spec and hook, such that pluggy will run them in serial?

Answer 1

As you also observed, there can be exactly one plugin with a specific name. When using entry points, the name of the plugin is the name of the entry point.

As far as I understand the docs, hooks are matched by the spec/impl name and signature, and the plugin name does not matter in this process (plugins are just named collections of related hook impls).

From these two, it follows that the entry point name should match the plugin name (and should be unique); see the plugin setup.py example. So, the pyproject.toml entry point would look something like a = "a".

(An unrelated possible issue with the code above is that the spec should be added to the manager with add_hookspecs(), example. Given your code works without it, it's probably not mandatory, but it allows the manager to validate the impls against the spec, and all the examples seem to use it.)