Python GIL and threads

Question

I have embedded Python3 in my big C++ application. Python gives the user script capability for custom data processing.
Problem : I have many threads that interact with Python and I don't really get how to protect my code with GIL. So far, the only way I made my code work is using boost::mutex.

Here is a very simplified example that reproduces my problem:

• Thread A calls first Init() to initialize Python (static function).
• Thread B calls Pythonize() to do some work on Python. Thread B is blocked on the first call for locking GIL.

Code:

#include <iostream>
#include <boost/thread.hpp>
#include <boost/bind.hpp>

#include "Python.h"

struct RTMaps_GILLock
{
    RTMaps_GILLock()
    {
        std::cout << "Locking..." << std::endl;
        m_state = PyGILState_Ensure();
    }

    ~RTMaps_GILLock()
    {
        std::cout << "Unlocking..." << std::endl;
        PyGILState_Release(m_state);
    }

private:
    PyGILState_STATE m_state;
};
#define GILLOCK RTMaps_GILLock lock;

class PythonEmbed
{
public:
    static void Init()
    {
        Py_Initialize();
        // EDIT : adding those two lines made my day :
        PyEval_InitThreads(); // This acquires GIL
        PyEval_SaveThread(); // Release the GIL
    }

    void Pythonize()
    {
        GILLOCK;
        // Never goes here :(
        std::cout << "OK" << std::endl;
    }
};

int main()
{
    PythonEmbed::Init();

    PythonEmbed pyt;
    boost::thread t(boost::bind(&PythonEmbed::Pythonize, pyt));

    t.join();
}

it is deadlocking in the first lock call. The console shows: Locking...

The "OK" is never printed. What am I doing wrong ?

EDIT : corrected code, now it is working. I needed to release the GIL from the main thread.

Answer 1

I had your exact problem, be sure to not call PyGILState_Ensure() from the main thread, the one that initialize Pythons, because it needs a total different call. I've end setting a thread mapper, and each call to my acquirePython() checks what thread is calling it, if it's the main thread , it uses:

PyEval_SaveThread();

otherwise it stores the GIL. Those are the relevant sections of my class:

void MManager::acquirePython(void) {
    MThread thisThread = MFramework::MProcesses::GetCurrentThread();
    if (thisThread != mainThread) {
        Lock();
        std::map<MThread,void*>::iterator i = threadStates.find(thisThread);
        if (i == threadStates.end()) {
            Unlock();
            PyGILState_STATE gstate = PyGILState_Ensure();
            _PyGILState_STATE_* encState = new _PyGILState_STATE_;
            encState->state = gstate;
            encState->refCount = 1;
            Lock();
            threadStates[thisThread] = encState;
            Unlock();
        } else {
            _PyGILState_STATE_* encState = (_PyGILState_STATE_*)i->second;
            encState->refCount = encState->refCount + 1;
            Unlock();
        }

    } else {
        if (mainThreadState) PyEval_RestoreThread((PyThreadState*)mainThreadState);
    }

}

void MManager::releasePython(void) {
    MThread thisThread = MFramework::MProcesses::GetCurrentThread();
    if (thisThread != mainThread) {
        Lock();
        std::map<MThread,void*>::iterator i = threadStates.find(thisThread);
        if (i != threadStates.end()) {
            _PyGILState_STATE_* encState = (_PyGILState_STATE_*)i->second;
            if (encState->refCount <= 1) {
                threadStates.erase(i);
                Unlock();

                PyGILState_Release(encState->state);
                delete encState;
            } else {
                encState->refCount = encState->refCount - 1;
                Unlock();
            }
        } else {
            Unlock();
        }

    } else {
        mainThreadState = PyEval_SaveThread();
    }
}