Calling python function from C as a callback. What is the right way to handle the GIL?

Question

I'm using cytpes to wrap a C api. One of the api functions allows you to register a callback. I'm using CFUNCTYPE to to specify the function's type and make an instance of CFUNCTYPE from a python function that the user of my python library provides which I then pass to the C function (called with the ctypes api).

I know that ctypes calls release the GIL. I'm wondering what happens when the C library function calls my python callback function. Does ctypes reacquire the GIL?

The documentation says:

Note: Make sure you keep references to CFUNCTYPE() objects as long as they are used from C code. ctypes doesn’t, and if you don’t, they may be garbage collected, crashing your program when a callback is made. Also, note that if the callback function is called in a thread created outside of Python’s control (e.g. by the foreign code that calls the callback), ctypes creates a new dummy Python thread on every invocation. This behavior is correct for most purposes, but it means that values stored with threading.local will not survive across different callbacks, even when those calls are made from the same C thread.

It doesn't say anything about the GIL. Does that mean that it's all handled for me?

Answer 1

The CThunkObject referenced in the callback's _objects attribute has the pcl_exec function pointer that the C library calls. This code calls closure_fcn with a reference to the thunk, plus the call args and a pointer to memory to store the result. The closure function in turn calls _CallPythonObject, with the thunk's restype, setfunc, callable, converters, and flags as parameters. The first thing _CallPythonObject does is call PyGILState_Ensure to acquire the GIL. This works even if this is the first time Python has seen the current thread.

In other words, it's all handled for you. Just keep a reference to the callback to keep the thunk referenced.