Fatal error in extension: PyThreadState_Get: no current thread

Question

I've seen several posts that have stated the same error, but looking and trying out the answers in those posts have not helped. I was wondering if someone could look at this and see if something pops out?

I'm building a Python extension for a CPP application, and there are no errors during the compilation and build step. However, when I import the module I get the error mentioned in the title. Other stackoverflow answers have claimed that this is because of being linked with one library while compilation and using a different interpreter. As far as I can tell, I'm using the same Python interpreter. I'm going to describe now why I think I'm using the same Python in the linking process and for the interpreter.

This is the comand I'm using to build the Python extension

$ gcc -shared helicsPYTHON_wrap.c $(python-config3 --includes) -I/path/to/helics-0.9/includes -L/path/to/helics-0.9/lib -lhelicsSharedLib -L$(python3-config --prefix)/lib -lpython3.6m -o _helics.so

$ which python3-config
/Users/$USER/miniconda3/bin/python3-config

$ python3-config --prefix
/Users/$USER/miniconda3

If I try to import the python file that imports the shared library, it throws the fatal error. If I use otool -L on the shared library, I get the following. This is what I expect to get.

$ otool -L _helics.so
_helics.so:
        @rpath/libhelicsSharedLib.dylib (compatibility version 0.0.0, current version 0.0.0)
        @rpath/libpython3.6m.dylib (compatibility version 3.6.0, current version 3.6.0)
        /usr/local/opt/zeromq/lib/libzmq.5.dylib (compatibility version 7.0.0, current version 7.3.0)
        libboost_program_options.dylib (compatibility version 0.0.0, current version 0.0.0)
        libboost_filesystem.dylib (compatibility version 0.0.0, current version 0.0.0)
        libboost_system.dylib (compatibility version 0.0.0, current version 0.0.0)
        libboost_date_time.dylib (compatibility version 0.0.0, current version 0.0.0)
        /usr/local/opt/gcc/lib/gcc/7/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.24.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)
        /usr/local/lib/gcc/7/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

I also tried install_name_tool to add the full path of the libpython3.6m.dylib.

$ install_name_tool -change @rpath/libpython3.6m.dylib /Users/$USER/miniconda3/envs/py3/lib/libpython3.6m.dylib _helics.so

I still get the same fatal error. My hypothesis is that my Mac System Python 2.7 installation is having an effect on this process at some stage. I'm unable to identify where though.

Is there a way to add more debug statements to find out why there is a Fatal Python error. Currently, the error message is very short.

$ python helics.py
Fatal Python error: PyThreadState_Get: no current thread

[1]    64481 abort      python helics.py

Curiously, if I use a conda environment and use Python 2.7, I'm able to load the extension fine! This is why I think that when I'm using Python 3.6, it is somehow picking up something from the default mac system python 2.7 installation and working fine. It is picking the same thing up when I use the conda 2.7 python environment, but because they are both Python 2.7 (though conda is 2.7.14 and system python is 2.7.10) it seems to work. This is the otool -L output when I use a conda environment.

$ otool -L _helics.so
_helics.so:
        @rpath/libhelicsSharedLib.dylib (compatibility version 0.0.0, current version 0.0.0)
        @rpath/libpython2.7.dylib (compatibility version 2.7.0, current version 2.7.0)
        /usr/local/opt/zeromq/lib/libzmq.5.dylib (compatibility version 7.0.0, current version 7.3.0)
        libboost_program_options.dylib (compatibility version 0.0.0, current version 0.0.0)
        libboost_filesystem.dylib (compatibility version 0.0.0, current version 0.0.0)
        libboost_system.dylib (compatibility version 0.0.0, current version 0.0.0)
        libboost_date_time.dylib (compatibility version 0.0.0, current version 0.0.0)
        /usr/local/opt/gcc/lib/gcc/7/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.24.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)
        /usr/local/lib/gcc/7/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

The questions I have are 1) how do I get more debug information out of the error from Python. I have tried python -vvv and that does not give me enough information. I tried using gdb but that also did not give me any information. I believe it requires recompiling Python itself using debug symbols. 2) Do you have any recommendations on how to solve this problem or debug further?

Also, I'm not sure if this is useful information, but I am able to use ctypes and load the shared library after I create it. I'm just unable to import it as a python module.

This is the original issue if one is interested - https://github.com/GMLC-TDC/HELICS-src/issues/59

Edit: I tried this using zsh and bash, and still got the same error. I also tried setting the following export PATH="/Users/$USER/miniconda3/bin:/Users/$USER/miniconda3/lib" temporarily in the shell and running and I STILL get the same error. This should have excluded my Mac System Python 2.7.10, so I'm really not sure what is going on.

Edit again: I've also tried reinstalling miniconda with Python2. And if I use Python2 everything works fine. I'm just unable to use Python3 using miniconda. Oddly enough, if I use homebrew and install Python3 that seems to work fine.

Edit again: This is possibly an issue with High Sierra. I currently don't have access to another mac, but I'm on the latest operating system which has SIP. I'm not sure if this is causing this issue. Additionally, I've tried using Anaconda3 and had no luck.

Edit again: This does not seem to be related to the operating system. I'm able to run this successfully on another computer with High Sierra.

Edit again: I tested this on other fresh OS installs, and they don't work. But they do work on two of my machines. Are there other tools that tell you what dependency a library requires or where Python throws a fatal error? My best guess at the moment is that I've installed something on my other machines in the past that allows this to work. I need to identify what that was and make sure I can document it.

Edit again: I've added a gist of the output of the version of Python that I'm using.

Edit again: I've added the tags for miniconda and anaconda since I don't experience this issue when using homebrew python3, but only just when I'm using miniconda3 or anaconda2 with a python3 environment. This always appears to work with Python2, regardless of whether it is homebrew, anaconda or miniconda.

Edit again:

These are the steps if someone else wants to replicate on their machine.

git clone https://github.com/GMLC-TDC/HELICS-src
mkdir build-osx
brew install boost
brew install cmake
brew install swig
cmake -DBUILD_PYTHON=ON -DPYTHON_LIBRARY=$(python3-config --prefix)/lib/libpython3.6m.dylib -DPYTHON_INCLUDE_DIR=$(python3-config --prefix)/include/python3.6m/ ..
make
cd ./swig/python/
python helics.py # Error

Answer 1

You have Python 3.6, from homebrew... While your module, while building referenced Python 2.7, provided by the system. Here is same problem described. From one of the comments - python3.6-config --ldflags will show LDFLAGS to be used in your Makefile.

Answer 2

I was able to solve this problem by changing the CMakeLists.txt to use -undefined dynamic_lookup as suggested in this answer. E.g. of the CMakeLists.txt is here. And the reason I was getting different outcomes on different machines was because one of my macs had Python 3.6.1 but the others had Python>=3.6.2