Calling Py_Finalize() from C

Question

This is a follow up to Call Python from C++

At the startup of the programm I call the following function to initialize the interpreter:

void initPython(){
    PyEval_InitThreads();
    Py_Initialize();
    PyEval_ReleaseLock();
}

Every thread creates it's own data structure and acquires the lock with:

PyGILState_STATE gstate;
gstate = PyGILState_Ensure();
//call python API, process results
PyGILState_Release(gstate);

Rather straight forward once you understood the GIL, but the problem is that I get a segfault when calling Py_Finalize().

void exitPython(){
    PyEval_AcquireLock();
    Py_Finalize();
}

The reference is rather dubious about Py_Finalize() (or maybe I'm just reading it the wrong way) and I'm not sure if PyEval_AcquireLock() can acquire the lock if there are some active threads and what happens if there are active threads when Py_Finalize() is called.

Anyways, I get a segfault even if I'm sure that all threads have finished their work, but only if at least one was created. E.g. calling initPython() followed from exitPython() creates no error.

I could just ignore the problem and hope the OS knows what it does, but I'd prefere if I could figure out what is going on..

Answer 1

Yeah the whole section is rather dubious but I think I've got my mistake.

I've got to save the PyThreadState when initializing the interpreter and swap this state back in when I finish it (no idea why I need a specific ThreadState to call Finalize - shouldn't every State work as well?)

Anyways the example if other people got the same problem:

PyThreadState *mainstate;

void initPython(){
    PyEval_InitThreads();
    Py_Initialize();
    mainstate = PyThreadState_Swap(NULL);
    PyEval_ReleaseLock();
}

void exitPython(){
    PyEval_AcquireLock();
    PyThreadState_Swap(mainstate);
    Py_Finalize();
}

The only problem with this is, that I can acquire the lock like every other thread, even if there are still threads working. The API doesn't mention what happens when Finalize() is called while other threads are still working. Sounds like the perfect example of a race condition..

Answer 2

Have you tried commenting out all the 'work' done in your threads? Replace it with a busy loop or a sleep or something. That will help pinpoint whether it is your initialisation/shutdown code, or something you're actually doing to Python in between. Perhaps you're not setting up the threads properly - there are a lot of thread-specific functions in the C API and I'm not sure which ones you need to ensure proper operation.

Answer 3

I am also getting similar problem while running scripts containing pyxhook from different threads through embedded interpreter.

There is no problem if one script running at a time. The hook is released properly but if two or more script ran in parallel the hooking is not stopped. Though my scripts returned properly and cancel() from pyxhook also returned properly, I think still some threads are running related to xlib. This pyxhook problem I have solved by keeping a global flag to watch if pyxhook is already running and not re-initializing pyxhook from every thread.

Now regarding Py_Finalize(), if pyxhook is re-initialized in every thread:

if I do not call PyEval_AcquireLock() and PyThreadState_Swap() before calling Py_Finalize() it terminates in Linux but not in Win32. In Win32 there is a problem if I do not go through PyEval_AcquireLock() and PyThreadState_Swap().

For the time being the temporary solution for me is to terminate differently in two different OS.