Is it possible to multiprocess a function that returns something in Python?

Question

In Python I have seen many examples where multiprocessing is called but the target just prints something. I have a scenario where the target returns 2 variables, which I need to use later. For example:

def foo(some args):    a = someObject    b = someObject    return a,b  p1=multiprocess(target=foo,args(some args)) p2=multiprocess(target=foo,args(some args)) p3=multiprocess(target=foo,args(some args)) 

Now what? I can do .start and .join, but how do I retrieve the individual results? I need to catch the return a,b for all the jobs I execute and then work on it.

Answer 1

You are looking to do some embarrassingly parallel work using multiple processes, so why not use a Pool? A Pool will take care of starting up the processes, retrieving the results, and returning the results to you.

I use pathos, which has a fork of multiprocessing, because it has much better serialization than the version that standard library provides.

(.py) file

from pathos.multiprocessing import ProcessingPool as Pool  def foo(obj1, obj2):     a = obj1.x**2     b = obj2.x**2     return a,b  class Bar(object):     def __init__(self, x):         self.x = x  Pool().map(foo, [Bar(1),Bar(2),Bar(3)], [Bar(4),Bar(5),Bar(6)]) 

Result

[(1, 16), (4, 25), (9, 36)] 

And you see that foo takes two arguments, and returns a tuple of two objects. The map method of Pool submits foo to the underlying processes and returns the result as res.

You can get pathos here: https://github.com/uqfoundation