Bug in Python? threading.Thread.start() does not always return

Question

I have a tiny Python script which (in my eyes) makes threading.Thread.start() behave unexpectedly since it does not return immediately.

Inside a thread I want to call a method from a boost::python based object which will not return immediately.

To do so I wrap the object/method like this:

import threading
import time
import my_boostpython_lib

my_cpp_object = my_boostpython_lib.my_cpp_class()

def some_fn():
    # has to be here - otherwise .start() does not return
    # time.sleep(1)  
    my_cpp_object.non_terminating_fn() # blocks

print("%x: 1" % threading.get_ident())
threading.Thread(target=some_fn).start()
print("%x: 2" % threading.get_ident())  # will not always be called!!

And everything works fine as long as I run some code before my_cpp_object.non_terminating_fn(). If I don't, .start() will block the same way as calling .run() directly would.

Printing just a line before calling the boost::python function is not enough, but e.g. printing two lines or calling time.sleep() makes start() return immediately as expected.

Can you explain this behavior? How would I avoid this (apart from calling sleep() before calling a boost::python function)?

Answer 1

This behavior is (as in most cases when you believe in a bug in an interpreter/compiler) not a bug in Python but a race condition covering the behavior you have to expect because of the Python GIL (also discussed here).

As soon as the non-Python function my_cpp_object.non_terminating_fn() has been started the GIL doesn't get released until it returns and keeps the interpreter from executing any other command.

So time.sleep(1) doesn't help here anyway because the code following my_cpp_object.non_terminating_fn() would not be executed until the GIL gets released.

In case of boost::python and of course in case you can modify the C/C++ part you can release the GIL manually as described here.

A small example (from the link above) could look like this (in the boost::python wrapper code)

class scoped_gil_release {
public:
    inline scoped_gil_release() {
        m_thread_state = PyEval_SaveThread();
    }

    inline ~scoped_gil_release() {
        PyEval_RestoreThread(m_thread_state);
        m_thread_state = NULL;
    }

private:
    PyThreadState * m_thread_state;
};

int non_terminating_fn_wrapper() {
    scoped_gil_release scoped;
    return non_terminating_fn();
}