PyGILState_Ensure() Causing Deadlock

Question

I'm writing a Python extension in C++, wrapping a third-party library I do not control. That library creates a thread Python knows nothing about, and from that thread, calls a C++ callback I provide to the library. I want that callback to call a Python function, but I get a deadlock using the approach I read from the docs. Here's my interpretation of those.

void Wrapper::myCallback()
{
   PyGILState_STATE gstate=PyGILState_Ensure();
   PyObject *result=PyObject_CallMethod(_pyObj,"callback",nullptr);
   if (result) Py_DECREF(result);
   PyGILState_Release(gstate);
}

My code does nothing else related to threads, though I've tried a number of other things that have. Based on this, for example, I tried calling PyEval_InitThreads(), but it's not obvious where that call should be made for an extension. I put it in the PyMODINIT_FUNC. These attempts have all lead to deadlock, crashes, or mysterious fatal errors from Python, e.g., PyEval_ReleaseThread: wrong thread state.

This is on Linux with Python 3.6.1. Any ideas how I can get this "simple" callback to work?

Likely Culprit

I didn't realize that in another thread, the library was in a busy/wait loop waiting on the callback's thread. In gdb, info threads made this apparent. The only solution I can see is to skip those particular calls to the callback; I don't see a way to make them safe, given the busy/wait loop. In this case, that's acceptable, and doing so eliminates the deadlock.

Also, it appears that I do need to also call PyEval_InitThreads() before any of this. In a C++ extension, it's not clear where that should go though. One of the replies suggested doing it indirectly in Python by creating and deleting a throwaway threading.Thread. That didn't seem to fix it, triggering instead a Fatal Python error: take_gil: NULL tstate, which I think means there's still no GIL. My guess, based on this and the issue it refers to, is that PyEval_InitThreads() causes the current thread to become the main thread for the GIL. If that call is made in the short-lived throwaway thread, maybe that's a problem. Yeah, I'm only guessing and would appreciate an explanation from someone who doesn't have to.

Answer 1

This answer is only for Python >= 3.0.0. I don't know if it would work for earlier Pythons or not.

Wrap your C++ module in a Python module that looks something like this:

import threading
t = threading.Thread(target=lambda: None, daemon=True)
t.run()
del t
from your_cpp_module import *

From my reading of the documentation, that should force threading to be initialized before your module is imported. Then the callback function you have written up there should work.

I'm less confident of this working, but your module init function could instead do this:

if (!PyEval_ThreadsInitialized())
{
    PyEval_InitThreads();
}

that should work because your module init function should be being executed by the only Python thread in existence if PyEval_ThreadsInitialized() isn't true, and holding the GIL is the right thing to do then.

These are guesses on my part. I've never done anything like this as is evidenced by my clueless comments on your question. But from my reading of the documentation, both of these approaches should work.

Answer 2

I'm new to StackOverflow, but I've been working on embedding python in a multithreaded C++ system for the last few days and run into a fair number of situations where the code has deadlocked itself. Here's the solution that I've been using to ensure thread safety:

class PyContextManager {
   private:
      static volatile bool python_threads_initialized;
   public:
      static std::mutex pyContextLock;
      PyContextManager(/* if python_threads_initialized is false, call PyEval_InitThreads and set the variable to true */);
      ~PyContextManager();
};

#define PY_SAFE_CONTEXT(expr)                   \
{                                               \
   std::unique_lock<std::mutex>(pyContextLock); \
   PyGILState_STATE gstate;                     \
   gstate = PyGILState_Ensure();                \
      expr;                                     \
   PyGILState_Release(gstate);                  \
}

Initializing the boolean and the mutex in the .cpp file.

I've noticed that without the mutex, the PyGILState_Ensure() command can cause a thread to deadlock. Likewise, calling PySafeContext within the expr of another PySafeContext will cause the thread to brick while it waits on its mutex.

Using these functions, I believe your callback function would look like this:

void Wrapper::myCallback()
{
   PyContextManager cm();
   PY_SAFE_CONTEXT(
       PyObject *result=PyObject_CallMethod(_pyObj,"callback",nullptr);
       if (result) Py_DECREF(result);
   );
}

If you don't believe that your code is likely to ever need more than one multithreaded call to Python, you can easily expand the macro and take the static variables out of a class structure. This is just how I've handled an unknown thread starting and determining whether it needs to start up the system, and dodging the tedium of writing out the GIL functions repeatedly.

Hope this helps!