Python multiprocessing - Passing a list of dicts to a pool

Question

This question may be a duplicate. However, I read lot of stuff around on this topic, and I didn't find one that matches my case - or at least, I didn't understood it.

Sorry for the inconvenance.

What I'm trying to do is fairly common, passing a list of kwargs to pool.starmap(), to achieve multiprocessing.

Here's my reduced case:

def compute(firstArg, **kwargs): # A function that does things
    # Fancy computing stuff...
    # Saving to disk...
    return True

if __name__ ==  '__main__':
    args = [{
        'firstArg': "some data",
        'otherArg': "some other data"
        'andAnotherArg': x*2
    } for x in range(42)]

    pool = Pool(4)
    pool.starmap(compute, args)
    pool.close()
    pool.terminate()

I supposed starmap() will unpack the dict and pass it to compute() as keyword args, but looking at the source code (see also l.46), it sends only keys (or values ?).

So it raises :

TypeError: compute() takes 1 positional argument but 3 were given

It must be a clear, straight forward way to do this... Any help would be appreciated.

Here's a quite similar question : Python Multiprocessing - How to pass kwargs to function?

Answer 1

You could use a tiny proxy function:

def proxy(Dict):
    return compute(**Dict)

pool.map(proxy, args)

Or, since you don't need the proxy function polluting the namespace:

pool.map(lambda Dict: compute(**Dict), args)