python child process crashes on numpy dot if PySide is imported

Question

I've got this very peculiar hanging happening on my machine when using pytnon multiprocessing Pool with numpy and PySide imported. This is the most entangled bug I have seen in my life so far:) The following code:

import numpy as np
import PySide


def hang():
    import multiprocessing
    pool = multiprocessing.Pool(processes = 1)
    pool.map(f, [None])


def f(ignore):
    print('before dot..')
    np.dot(np.zeros((128, 1)), np.zeros((1, 32)))
    print('after dot.')


if __name__ == "__main__":
    hang()
    print('success!')

hangs printing only 'before dot..'. But it is supposed to print

before dot..
after dot.
success!

I'm not gdb expert, but looks like gdb shows that processes exits (or crashes) on 'np.dot' line:

[Inferior 1 (process 2884) exited normally]

There are several magical modifications I can do to prevent hanging:

• if you decrease shape of arrays going into 'dot' (e.g. from 128 to 127)
• (!) if you increase shape of arrays going into 'dot' from 128 to 256
• if you do not use multiprocessing and just run function 'f'
• (!!!) if you comment out PySide import which is not used anywhere in the code

Any help is appreciated!

Packages version:

numpy=1.8.1 or 1.7.1 PySide=1.2.1 or 1.2.2

Python version:

Python 2.7.5 (default, Sep 12 2013, 21:33:34) [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin

or

Python 2.7.6 (default, Apr 9 2014, 11:48:52) [GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.38)] on darwin

Notice: While hunting for a information, I simplified original code and question a bit. But here is a stack of updates to keep history for others who may encounter this bug (e.g. I started with matplotlib, not with pyside)

Update: I narrowed down pylab import to importing matplotlib with pyside backend and updated the code to run.

Update: I'm modifying the post to import only PySide only instead of:

import matplotlib
matplotlib.use('qt4agg')
matplotlib.rcParams['backend.qt4']='PySide'
import matplotlib.pyplot

Update: Initial statistics shows that it is a Mac-only issue. 3 people have it working on Ubuntu, 2 people got it hanging on Mac.

Update: print(os.getpid()) before dot operation gives me pid that I don't see in 'top' that apparently means that it crashes and multiprocessing waits for a dead process. For this reason I can not attach debugger to it. I edited main question accordingly.

Answer 1

this is a general issue with some BLAS libraries used by numpy for dot.

Apple Accelerate and OpenBlas built with GNU Openmp are known to not be safe to use on both sides of a fork (the parent and the child process multiprocessing create). They will deadlock.

This cannot be fixed by numpy but there are three workarounds:

• use netlib BLAS, ATLAS or git master OpenBlas based on pthreads (2.8.0 does not work)
• use python 3.4 and its new multiprocessing spawn or forkserver start methods
• use threading instead of multiprocessing, numpy releases the gil for most expensive operations so you can archive decent threading speedups on typical desktop machines