Embedding multiple Python sub-interpreters into a C program

Question

I am writing a C program that spawns multiple C threads, with one Python sub-interpreter per thread. The sub-interpreters do not share any mutable Python variables, they are isolated from each other. (They do have a read-only access to a common PyObject (immutable) that is exposed from the main() function in the C program).

Is this possible in Python 3.7 or 3.8, without sharing GIL between the sub-interpreters?

Here is the pseudo-code of what I have been trying:

void *spawnInterpreter(void* p) {
    …
    PyThreadState* save_tstate = PyThreadState_Swap(NULL);
    PyThreadState* tstate = Py_NewInterpreter();
    PyThreadState_Swap(save_tstate);

    //do some Python work (with variables that are NOT shared with other thread’s sub-interpreter
    PyRun_SimpleString( . . .);
    . . . 
}


int main() {
...
    pthread_create(&thread1, NULL, spawnInterpreter,  “in1”);
    pthread_create(&thread2, NULL, spawnInterpreter, "in2");
...
}

I could get this to work in 3.6 (without acquiring GIL or managing PyThreadState in C threads), but in Python 3.7 I get:

[New Thread 0x7ffff5f78700 (LWP 16392)]
Fatal Python error: drop_gil: GIL is not locked

Answer 1

Unfortunately, subinterpreters still share the GIL in 3.7 and 3.8. This is something I'm personally working on changing. See PEP 554 and my multi-core Python project. I'm also giving a talk at PyCon next week that covers the topic in some detail.

My hope has been to make it possible in Python 3.8, but it's looking more likely for 3.9 at this point. The main challenge is that the C-API and CPython runtime are not thread-safe. While most of the C-API and runtime can switch to using the per-interpreter GIL, other things will have to change in that scenario:

• some process-global resources have to be managed more carefully without a GIL (e.g. env vars, file handles)
• there is global runtime state that interpreters must continue to share, so that much must still be protected by a global lock (though one which will not need to block the Python bytecode eval loop)
• some of the global runtime state needs to be moved down to per-interpreter state (e.g. GC, memory allocators, warnings)
• objects will need to be strictly per-interpreter (for now), so the C-API must be strict in not allowing objects to cross the interpreter boundary
• parts of the C-API that are not specific to an interpreter context must change to no longer require holding the GIL

The problem is tractable, but it takes time to apply the necessary care when working on such critical code. Hence the likely target of 3.9.

---

Regardless, I'm thankful that you've posted here. Most of my efforts have focused on the impact on Python code, rather than the C-API (e.g. embedders). So feedback on how my project relates to use of subinterpreters via the C-API is super helpful. For instance, one thing you've reminded me of is that creating subinterpreters via the C-API is slightly different than the equivalent in PEP 554. That needs to be considered more carefully. Also, PEP 554 exposes virtually none of its additions in the C-API. That's probably okay, but interacting with channels from the C-API might be valuable in the short term.