Splitting up pybind11 modules and issues with automatic type conversion

Question

I have a set of modules that I've written in C++ and exported to Python using pybind11. All of these modules should be able to be used independently, but they use a common set of custom types that are defined in a utility library.

In each module there is code something like what's below. The Color.hpp header defines the types that are used in the utility library.

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <string>
#include "Color.hpp"

std::vector<Color> buncha_colors(int n, std::string &color) {
    std::vector<Color> out;
    for (;n-- > 0;) {
        out.push_back(Color(color));
    }
    return out;
}

PYBIND11_MODULE(pb11_example_module, m) {
    m.def("buncha_colors", &buncha_colors);
}

Of course, this doesn't work. Pybind does not know how to do the type conversion for a Color object. The answer (or hopefully not) is to define the Color class as a part of the module. Afterwards, pybind is capable of doing the automatic type conversion.

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <string>
#include "Colors.hpp"

std::vector<Color> buncha_colors(int n, std::string &color) {
    std::vector<Color> out;
    for (;n-- > 0;) {
        out.push_back(Color(color));
    }
    return out;
}

PYBIND11_MODULE(pb11_example_module, m) {
    pybind11::class_<Color>(m, "Color")
        .def(pybind11::init<std::string&>())
        .def("name", &Color::name);
    m.def("buncha_colors", &buncha_colors);
}

Ideally, I'd like to keep all these custom utility types and associated functions in a separate module from all the modules that use them. But I'd need to define the type conversions, or otherwise reference it, in every module that uses it. How do I go about this? I don't want pb11_example_module.Color and utils.Color and so on. I have no idea about their compatibility for one, and it just seems like the wrong way.

Answer 1

This started out as an edit, but has turned out to be my answer.

So this is interesting. Using the first example where Color is not exported in the pybind module...

$ python
>>> import pb11_example_module
>>> pb11_example_module.buncha_colors(10, "red")[0].name()
TypeError: Unable to convert function return value to a Python type! The signature was
    (arg0: int, arg1: str) -> List[Color]
>>> import utils  # defines Color
>>> pb11_example_module.buncha_colors(10, "red")[0].name()
'red'

Importing the utility library before importing the example module also works. Changing the class name "Color" to something else doesn't break the usage either, so it must be picking up the type conversion from the other module using the type signature.

As long as a type conversion for that C++ type is defined before use, automatic type conversion works. The type conversion utilities that pybind uses for Python are global and looked-up at runtime. You can read all about it here. The above solution of loading a type conversion for the custom type at any point before usage is the supported idiomatic solution.