Python code calls C library that create OS threads, which eventually call Python callbacks

Question

If the one and only Python interpreter is in the middle of executing a bytecode when the OS dispatches another thread, which calls a Python callback - what happens? Am I right to be concerned about the reliability of this design?

Answer 1

In general situations, the C library needs to call PyEval_InitThreads() to gain GIL before spawning any thread that invokes python callbacks. And the callbacks need to be surrounded with PyGILState_Ensure() and PyGILState_Release() to ensure safe execution.

However, if the C library is running within the context of, say, a python C extension, then there are simple cases where it is safe to omit GIL manipulation at all.

Think of this calling sequence: 1) python code calls C function foo(), 2) foo() spawns one and only one thread that runs another C function bar(), which calls back to python code, and 3) foo() always joins or cancels the thread running bar() before returns.

In such a case, it is safe to omit GIL manipulation. Because foo() owns the GIL (i.e. implicitly borrowed from the python code that calls it) during its lifecycle, and the execution of python callbacks within the lifecycle of foo() is serialized (i.e. only one callback thread and python code does not incorporate threading).