Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Multiprocessing Pool - how to cancel all running processes if one returns the desired result?

Given the following Python code:

import multiprocessing

def unique(somelist):
    return len(set(somelist)) == len(somelist)


if __name__ == '__main__':
    somelist = [[1,2,3,4,5,6,7,8,9,10,11,12,13,2], [1,2,3,4,5], [1,2,3,4,5,6,7,8,9,1], [0,1,5,1]]

    pool = multiprocessing.Pool()
    reslist = pool.map(unique, somelist)
    pool.close()
    pool.join()
    print "Done!"

    print reslist

Now imagine, that the lists with integers in this toy example are extremely long, and what I'd like to achieve here is the following: if one of the lists in somelist returns True, kill all running processes.

This leads to two questions (and probably more which I haven't come up with):

  • How can I "read" or "listen" from a finished process the result, while other processes are running? If e.g. a process is dealing with [1,2,3,4,5] from somelist, and is finished before all other processes, how can I read out the result from that process in this very moment?

  • Given the case that it is possible to "read" out the result of a finished process while other are running: how can I use this result as a condition to terminate all other running processes?

    e.g. If one process has finished and returned True, how I can use this as a condition to terminate all other (still) running processes?

like image 814
Daniyal Avatar asked Sep 10 '16 18:09

Daniyal


1 Answers

Use pool.imap_unordered to view the results in any order they come up.

reslist = pool.imap_unordered(unique, somelist)
pool.close()
for res in reslist:
    if res:  # or set other condition here
        pool.terminate()
        break
pool.join()

You can iterate over an imap reslist in your main process while the pool processes are still generating results.

like image 56
Tore Eschliman Avatar answered Nov 14 '22 23:11

Tore Eschliman