Python 3.4 multiprocessing recursive Pool.map()

Question

I'm developing with Python 3.4 on Ubuntu 14.04. I was trying to do recursive Pool.map(). After I invoke g(), it hangs there and never returns.

import multiprocessing as mp

pool = mp.Pool()

def d(x):
    return x / 2.0


def f(x):
    w = pool.map(d, x)
    return w

def g():
    v = pool.map(f, [[1, 2], [3, 4]])

    print(v)

Answer 1

From the documentation:

Note that the methods of the pool object should only be called by the process which created the pool.

You break that rule when you call pool.map() inside f(). Additionally, if you try to have the child process create their own pool, you'll get an assertion error:

AssertionError: daemonic processes are not allowed to have children

Answer 2

This isn't possible. The Pool object itself can't safely be shared between processes, so the same pool can't be used in both f and g. Even if you could do this, you'd quickly cause a hang, because your pool is limited to cpu_count() concurrent workers. Once you start creating more workers recursively, you'll end up with more than cpu_count() workers, which will never be able to finish; the running workers would be waiting on tasks that are queued up in the pool, but the queued up tasks won't ever be able to start because the running tasks are waiting. So you end up deadlocked. In short: don't try to do this.