python multiprocessing pool: how can I know when all the workers in the pool have finished?

Question

I am running a multiprocessing pool in python, where I have ~2000 tasks, being mapped to 24 workers with the pool. each task creates a file based on some data analysis and webservices.

I want to run a new task, when all the tasks in the pool were finished. how can I tell when all the processes in the pool have finished?

Answer 1

You want to use the join method, which halts the main process thread from moving forward until all sub-processes ends:

Block the calling thread until the process whose join() method is called terminates or until the optional timeout occurs.

from multiprocessing import Process

def f(name):
    print 'hello', name

if __name__ == '__main__':
    processes = []
    for i in range(10):
        p = Process(target=f, args=('bob',))
        processes.append(p)

    for p in processes:
        p.start()
        p.join()

     # only get here once all processes have finished.
     print('finished!')

EDIT:

To use join with pools

    pool = Pool(processes=4)  # start 4 worker processes
    result = pool.apply_async(f, (10,))  # do some work
    pool.close()
    pool.join()  # block at this line until all processes are done
    print("completed")