Python multiprocessing - Why does pool.close() take so long to return?

Question

Sometimes a call to the function pool.close() takes a lot of time to return, and I want to understand why. Typically, I would have each process return a big set or a big dict, and the main merge them. It looks like this:

def worker() :
    s = set()
    # add millions of elements to s
    return s

if __name__ == '__main__' :
    pool = multiprocessing.Pool( processes=20 )
    fullSet = set.union( * pool.imap_unordered( worker, xrange(100) ) )
    pool.close() # This takes a LOT OF TIME!
    pool.join()

As I said, the pool.close() might take 5, 10 min or more to return. Same problem occurs when using dictionaries instead of sets. This is what the documentation says about close:

Prevents any more tasks from being submitted to the pool. Once all the tasks have been completed the worker processes will exit.

I guess I don't understand what's going on. After the line fullSet = ..., all the work is done and I don't need the workers anymore. What are they doing that is taking so much time?

Answer 1

It is very unlikely that Pool.close is taking that long. Simply because this is the source of close

def close(self):
    debug('closing pool')
    if self._state == RUN:
        self._state = CLOSE
        self._worker_handler._state = CLOSE

So all that’s happening is that some state variables are changed. This has no measurable impact on the runtime of that method and will not cause it to return later. You could just assume close to return instantaneously.

Now instead, what’s way more likely is that your pool.join() line is the “culprit” of this delay. But it’s just doing its job:

Wait for the worker processes to exit.

It essentially calls join on every process in the pool. And if you are joining a process or thread, you are actively waiting for it to complete its work and terminate.

So in your case, you have 20 processes running that add a million elements to a set. That takes a while. To make your main process not quit early (causing child processes to die btw.), you are waiting for the worker processes to finish their work; by joining on them. So what you’re experiencing is likely what should happen for the amount of work you do.

On a side note: If you do heavy CPU work in your worker functions, you shouldn’t spawn more processes than your CPU has hardware threads available, as you will only introduce additional overhead from managing and switching processes. For example for a consumer Core i7, this number would be 8.