Why does PyGILState_Release(…) segfault in this case?

Question

I am working on implementing asynchronous audio playback for PyAudio. The backend Portaudio implements asynchronous playback by creating its own thread and calling a C-callback function whenever it needs/has new audio data. Whenever that C-callback function is called, I am calling a previously registered Python function where the user has to supply audio data.

Since this call to Python is happening in a non-Python-created thread, the documentation says I have to call PyGILState_Ensure() before the call to Python and PyGILState_Release() thereafter. It roughly looks like this:

int stream_callback(const void *in, void* out, unsigned long frameCount,
                    const PaStreamCallbackTimeInfo *timeInfo,
                    PaStreamCallbackFlags statusFlags, void *userData)
{
    PyGILState_STATE gstate = PyGILState_Ensure();

    /* create some python variables, as used below… */
    py_result = PyObject_CallFunctionObjArgs(py_callback,
                                             py_frameCount,
                                             py_inTime,
                                             py_curTime,
                                             py_outTime,
                                             py_inputData,
                                             NULL);
    /* evaluate py_result, do some audio stuff… */

    PyGILState_Release(gstate);
    return returnVal;
}

Which segfaults at PyGILState_Release(gstate). This callback function is invoked very often. Like, a few hundred to a few thousand times per second. The gstate is a 32 bit variable and is sometimes set to 1, sometimes to 0 by PyGILState_Ensure(). It only crashes when it is set to 1. Usually, there will be one 1 followed by two to four 0.

This kind of feels like this PyGILState_Release(…) is taking a bit longer than its actual return and thus is invoked while still running or something like this.

When crashing, the stack trace looks like this:

#0  0x00007fff88c287b7 in pthread_mutex_lock ()
#1  0x00000001001009a6 in PyThread_release_lock ()
#2  0x00000001002efc82 in stream_callback (in=0x1014a4670, out=0x1014a4670, frameCount=4316612208, timeInfo=0x1014a4850, statusFlags=4297757032, userData=0x38) at _portaudiomodule.c:1554
#3  0x00000001004e3710 in AdaptingOutputOnlyProcess ()
#4  0x00000001004e454b in PaUtil_EndBufferProcessing ()
#5  0x00000001004e9665 in AudioIOProc ()
#6  0x00000001013485d0 in dyld_stub_strlen ()
#7  0x0000000101348194 in dyld_stub_strlen ()
#8  0x0000000101346523 in dyld_stub_strlen ()
#9  0x0000000101345870 in dyld_stub_strlen ()
#10 0x000000010134aceb in AUGenericOutputEntry ()
#11 0x00007fff88aa132d in HP_IOProc::Call ()
#12 0x00007fff88aa10ff in IOA_Device::CallIOProcs ()
#13 0x00007fff88aa0f35 in HP_IOThread::PerformIO ()
#14 0x00007fff88a9ef44 in HP_IOThread::WorkLoop ()
#15 0x00007fff88a9e817 in HP_IOThread::ThreadEntry ()
#16 0x00007fff88a9e745 in CAPThread::Entry ()
#17 0x00007fff88c5c536 in _pthread_start ()
#18 0x00007fff88c5c3e9 in thread_start ()

Does this make sense to anyone?

Answer 1

I had the exact same issue. The fix was to call PyEval_InitThreads() on the main thread before any callbacks occur.

I believe the reason for this is as follows. When the Python interpreter first starts it avoids initializing the GIL because most Python programs are single-threaded and the presence of the GIL incurs some small performance penalty. So without the GIL being initialized PyGILState_Ensure() and PyGILState_Release() work on uninitialized data, causing odd crashes all over the place.

By calling PyEval_InitThreads() the GIL is initialized and PyGILState_Ensure() and PyGILState_Release() work correctly. If the GIL is already initialized PyEval_InitThreads() does nothing, so is safe to call over and over again.