Run many functions in parallel, and gather all results in a list

Question

I have a very CPU-intensive function:

def entity_intersections(ent, collidable):
    intersections = []

    for line1, line2 in product(ent.shape, collidable.shape):

        pair_intersections = find_intersections(line1 + ent.position, ent.velocity, ent.acceleration, line2 + collidable.position, collidable.velocity, collidable.acceleration, ent, collidable)
        intersections.extend(pair_intersections)

    return intersections

I want to make all of the calls to find_intersections run in parallel, so that they execute faster, while still gathering all of the results together (once all executions finish). What library would allow me to do this, given that find_intersections is a pure function?

An example of how to generate these parallel executions, as well as gathering together the results would be greatly appreciated.

Answer 1

The easiest way is to use the multiprocessing module:

class FindIntersectionsWrapper(object):
    def __init__(self, ent, collidable):
        self.ent = ent
        self.collidable = collidable
    def __call__(self, dims):
        line1, line2 = dims
        return find_intersections(
            line1 + self.ent.position, self.ent.velocity,
            self.ent.acceleration, line2 + self.collidable.position,
            self.collidable.velocity, self.collidable.acceleration, 
            self.ent, self.collidable)

def entity_intersections(ent, collidable):
    find_inter = FindIntersectionsWrapper(ent, collidable)
    pool = multiprocessing.Pool()
    return pool.map(find_inter, product(ent.shape, collidable.shape))

The helper function find_intersections_wrapper() is necessary since Pool.map() expects a function with a single argument.

You might want to move the creation of pool out of entity_intersections() to have the overhead of generating the process pool only once.

Edit: Used a class instead of a closure since the callable passed to Pool.map() must be picklable on Windows.