Using Python's multiprocessing module together with the cvxopt package

Question

I just came across an issue with the cvxopt package for convex optimization, which I didn't find mentioned in the documentation. I wonder if anybody knows what causes it and how to best work around it.

The issue is that when you import cvxopt in a Python program that makes use of multiprocessing.Process instances, the processes can no longer run in parallel. They seem to get automatically synchronized. Note that this happens regardless of whether any of the cvxopt functions are actually used by the program. Simply importing the package causes this effect.

Example:

# import cvxopt
from multiprocessing import Queue, Process

def compute(queue):
    """
    Pick integers from a queue and perform some useless
    calculations on them just to keep the CPU busy.
    """
    total = 0
    while True:
        item = queue.get()
        if item is None:
            break
        for i in range(item):
            total += i

if __name__ == '__main__':
    queue = Queue()
    procs = []
    for i in range(4):
        proc = Process(target = compute,
                       args   = (queue,))
        proc.start()
        procs.append(proc)

    for i in range(100000):
        queue.put(i)
    for proc in procs:
        queue.put(None)
    for proc in procs:
        proc.join()

The script above starts four processes running in parallel to the main program. On a machine with four cores I got four processes, each taking 100% of a CPU.

However, when uncommenting the import cvxopt statement at the start, each process ends up taking only 25% of a CPU, as if it was synchronized with the others.

I am interested in both explanations and work-arounds. I apologize if there is an obvious explanation in the documentation that I overlooked.

---

Details on how to reproduce the problem

The environment I am running this under is Linux 3.5.0 (Ubuntu 12.10 distribution), and Python 3.2.3. This is how I installed cvxopt:

1. Install matplotlib 1.2.1 (I did this because it includes pylab, which is used in many of the cvxopt example scripts. I am unsure if this is an actual requirement). I used this tar.gz package and did sudo python3 setup.py install to install it.

2. Installed BLAS and ATLAS

   sudo apt-get install libblas-dev libblas3 libatlas-base-dev libatlas3-base libblas-test libopenblas-base libopenblas-dev
   

   The above is probably more than necessary, but I'd be surprised if that caused any problems.

3. Installed cvxopt 1.1.6, using the tar.gz package from here and sudo python3 ./setup.py install.

Answer 1

Maybe something to do with the Python GIL not released ?

See : http://comments.gmane.org/gmane.comp.python.scientific.user/15678