boost.python not supporting parallelism?

Question

I am trying to wrap a piece of C++ code into python lib using boost.python, however, I found out that multiple instances cannot run at the same time:

code (C++):

class Foo{
public:
    Foo(){}
    void run(){
        int seconds = 2;
        clock_t endwait;
        endwait = clock () + seconds * CLOCKS_PER_SEC ;
        while (clock() < endwait) {}
    }   

};

BOOST_PYTHON_MODULE(run_test)
{
   using namespace boost::python;

   class_<Foo>("test", init<>())
      .def("run", &Foo::run)
      ;   

}

which is compile using CMake (CMake):

add_library(run_test SHARED run_test.cpp)
target_link_libraries(run_test boost_python python2.7)

and tested with the following code (Python):

class Dos(threading.Thread):
    def run(self):
        printl('performing DoS attack')

        proc = test()
        proc.run()

for i in range(5):
    t = Dos()
    t.start()

The output indicates that the code is parallelized in a very weird way. Each thread should take only 2 seconds and 4 threads should run simultaneously on my quadcore machine:

[2011-11-04 13:57:01] performing DoS attack
 [2011-11-04 13:57:01] performing DoS attack
[2011-11-04 13:57:05] performing DoS attack
 [2011-11-04 13:57:05] performing DoS attack
[2011-11-04 13:57:09] performing DoS attack

thank you for your help!

Answer 1

What you are running into is the python Global Interpreter Lock. The GIL only allows one thread at a time to run in the python interpreter.

One of the advantages of Boost.Python is that you can release the GIL, do C++ stuff, and then take it back when you are done. This is also a responsibility however. Python normally releases the GIL at regular intervals, to give other threads a chance to run. If you are in C++, this is your job. If you go crunch numbers for 2 hours while holding the GIL, you will freeze the whole interpreter.

This can be easy to fix with a little reverse RAII:

class releaseGIL{
public:
    inline releaseGIL(){
        save_state = PyEval_SaveThread();
    }

    inline ~releaseGIL(){
        PyEval_RestoreThread(save_state);
    }
private:
    PyThreadState *save_state;
};

Now you can change your code like so:

class Foo{
public:
    Foo(){}
    void run(){
        {
            releaseGIL unlock = releaseGIL();
            int seconds = 2;
            clock_t endwait;
            endwait = clock () + seconds * CLOCKS_PER_SEC ;
            while (clock() < endwait) {}
        }
    }   
};

It is VERY important to note that you MUST NOT touch any python code, or python data or call in to the interpreter while not holding the GIL. This will cause your interpreter to crash.

It is also possible to go the other way. A thread not currently holding the GIL can acquire it, and make calls in to python. This can be a thread that released the GIL earlier, or one that started in c++ and never had the GIL. Here is the RAII class for that:

class AcquireGIL 
{
public:
    inline AcquireGIL(){
        state = PyGILState_Ensure();
    }

    inline ~AcquireGIL(){
        PyGILState_Release(state);
    }
private:
    PyGILState_STATE state;
};

Usage is left as an exercise for the student.

Additional note (I always forget to mention this):

If you are going to be messing with the GIL in c++ your module definition needs to start with this code:

BOOST_PYTHON_MODULE(ModuleName)
{
    PyEval_InitThreads();

    ...
}