Is this Boost::Python (Python 3.7) error "__init__() should return None, not 'NoneType'" a linking problem?

Question

Update

I'm not going to add this as an answer, since I still haven't technically solved the problem. But since I've now spent 2.5 days trying to get things to work with boost-python3, I've lost the will to live with it.

I've just come across pybind11 (how my previous lengthy searches for python binding tools didn't turn it up, I don't know) and am using that. 2.5 days of misery compares to <20 minutes installing and building their cmake example... and all the specific-python-version-dependency-hell is gone.

It's syntactically similar to boost-python but much easier to manage, quicker, is header-only and is more feature rich.

Yay!

Original question

I'm using boost::python to bind a class in python 3.7.2.

The class import successfully but instantiating it gives the following error:

<my-terminal>$ python
Python 3.7.2 (default, Feb 14 2019, 17:36:47) 
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import classes
>>> t = classes.World()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: __init__() should return None, not 'NoneType'
>>> 

Here is classes.cpp:

#include <boost/python.hpp>
#include <boost/python/list.hpp>
#include <boost/python/extract.hpp>
#include <string>
#include <sstream>
#include <vector>

struct World
{
    void set(std::string msg) { mMsg = msg; }
    void many(boost::python::list msgs) {
        long l = len(msgs);
        std::stringstream ss;
        for (long i = 0; i<l; ++i) {
            if (i>0) ss << ", ";
            std::string s = boost::python::extract<std::string>(msgs[i]);
            ss << s;
        }
        mMsg = ss.str();
    }
    std::string greet() { return mMsg; }
    std::string mMsg;
};

using namespace boost::python;

BOOST_PYTHON_MODULE(classes)
{
    class_<World>("World")
        .def("greet", &World::greet)
        .def("set", &World::set)
        .def("many", &World::many)
    ;
};

Hypothesis

This question, almost identical was solved because of a python 2/3 issue (linking against python 3 instead of python 2 libraries). So I suspected a library linking issue.

Checking the hypothesis

I can't get bjam to work, and wouldn't be able to switch all our build systems over for one module anyway... so am building with cmake, which compiles successfully to classes.so with output as follows, suggesting I'm finding all the correct includes, libraries and executables:

-- Found PythonInterp: /Users/me/.pyenv/versions/boost37/bin/python3 (found suitable version "3.7.2", minimum required is "3") 
PYTHON_VERSION_SUFFIX
-- Boost version: 1.68.0
-- Found the following Boost libraries:
--   python37
-- Found PythonLibs: /usr/local/Frameworks/Python.framework/Versions/3.7/lib/libpython3.7m.dylib (found suitable version "3.7.2", minimum required is "3") 
-- PYTHON_LIBRARIES = /usr/local/Frameworks/Python.framework/Versions/3.7/lib/libpython3.7m.dylib
-- PYTHON_EXECUTABLE = /Users/thc29/.pyenv/versions/boost37/bin/python3
-- PYTHON_INCLUDE_DIRS = /usr/local/Frameworks/Python.framework/Versions/3.7/include/python3.7m
-- Boost_LIBRARIES = /usr/local/lib/libboost_python37-mt.dylib

Boost-python3 library directory contents:

ls /usr/local/Cellar/boost-python3/1.68.0/lib
libboost_numpy37-mt.a       libboost_numpy37.dylib      libboost_python37.a
libboost_numpy37-mt.dylib   libboost_python37-mt.a      libboost_python37.dylib
libboost_numpy37.a      libboost_python37-mt.dylib

I used brew install boost, and brew install boost-python3 --build-from-source with my python 3.7 virtualenv activated, to ensure boost-python3 is linked against the correct version of python.

Checking libraries...

otool -L classes.so gives:

classes.so:
    /usr/l/opt/boost-python3/lib/libboost_python37-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/Python (compatibility version 3.7.0, current version 3.7.0)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.4)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.200.5)

otool -L /usr/local/opt/boost-python3/lib/libboost_python37-mt.dylib gives:

/usr/local/opt/boost-python3/lib/libboost_python37-mt.dylib:
        /usr/local/opt/boost-python3/lib/libboost_python37-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.4)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.200.5)

In the related question, that showed their problem. But here it appears fine!

No progress yet...

After the painful process of getting this all compiling properly and checking the linking, I can't spot any flaws. Is this a different problem? Or is there a linking issue that I haven't spotted?

Thanks for any help!

Answer 1

Adding an answer here for those using the Anaconda or Conda-Forge Distribution:

The python interpreter statically links in the libpythonXY library. Which is why it makes the python binary different compared to other distributions.

The fix for the problem reported by OP is to use:

-undefined dynamic_lookup

Instead of:

-lpythonXY

You are creating a Python C/C++ extension, and not embedding the python interpreter. So you shouldn't be linking to the python library. Pybind11 handles this correctly.

See the following for more information:

• https://gitlab.kitware.com/cmake/cmake/issues/18100
• https://github.com/ContinuumIO/anaconda-issues/issues/9078

One a side note, python 3.8 has added an additional flag: --embed and only then it adds -lpythonXY in the output:

$ python3.8-config --libs
-ldl -framework CoreFoundation

$ python3.8-config --libs --embed
-lpython3.8 -ldl -framework CoreFoundation

Answer 2

I am following a similar example and I adopt the Makefile from here. I have installed python 3.7.4 and boost-python via brew on macOS. To fix the NoneType issue, I follow the procedure below:

1. Check the Python Path
To check the python path, use

which python

If the output does not look like the following one (brew's python installation path)

/usr/local/opt/python/libexec/bin/python

set the PATH variable as

export PATH="/usr/local/opt/python/libexec/bin:$PATH"

Check if the Python path looks like the one above again.

2. Check the Compilation Flag
Below is the adopted Makefile. Note the LIB variable. If the boost-python flag is -lboost_python, change it to -lboost_python37.

CPP = clang++
PYLIBPATH = $(shell python-config --exec-prefix)/lib
# LIB = -L$(PYLIBPATH) $(shell python-config --libs) -lboost_python
LIB = -L$(PYLIBPATH) $(shell python-config --libs) -lboost_python37
OPTS = $(shell python-config --include) -O2

default: hello.so


hello.so: hello.o
    $(CPP) $(LIB) -Wl,-rpath,$(PYLIBPATH) -shared $< -o $@

hello.o: hello.cpp Makefile
    $(CPP) $(OPTS) -c $< -o $@

clean:
    rm -rf *.so *.o

.PHONY: default clean

Recompile the C++ code and run the python script. The NoneType issue should disappear.

Hope this helps.

Note
If you are using anaconda and want to restore the PATH variable after the above changes, try

export PATH="~/anaconda3/bin:$PATH"

Your anaconda's path may be different.

Credit
1. George's comment in How do I use brew installed Python as the default Python?
2. leiyc's comment in ld: library not found for -lboost_python on MacOS