Python - Multiprocessing StarMap with two arguments

Question

I've a function that runs multiple queries in parallel but I'm having some troubles to run my function using multprocessing with more than argument. I've this code:

def run(args):
    query, cursor = args
    cursor.execute(query)
with multiprocessing.Pool(processes=10) as pool:
    args = (product(queries),cursor)
    results = pool.starmap(run(args))

If I run only pool.starmap(run(product(queries))) it works well, however I have the need to pass also the cursor object.

How I can do that?

I'm having the following error:

TypeError: starmap() missing 1 required positional argument: 'iterable'

Answer 1

There are some problems with your code:

• you call run with parameters, then pass the result to starmap, but you have to pass both the function and its parameters separately to starmap
• your args are a tuple of first the product of all queries, then the cursor, but you rather want to combine each of those query-combinations with the cursor
• your function expects a single parameter that you then unpack inside the function, so you should use map; for starmap it should be def run(query, cursor)

Try this:

import multiprocessing
import itertools

def run(args):
    query, cursor = args
    print("running", query, cursor)

queries = ["foo", "bar", "blub"]
cursor = "whatever"
    
with multiprocessing.Pool(processes=10) as pool:
    args = ((args, cursor) for args in itertools.product(queries))
    results = pool.map(run, args)

There may be more "upstream" errors, like that SSLSocket stuff, but this should at least (try to) call the function with the correct parameters.

Answer 2

pool.starmap takes two arguments: the function and a list of arguments.

From the docs:

pool.starmap(func, [(1,2), (3, 4)])
# results in [func(1,2), func(3,4)]
# I guess in your case would be

pool.starmap(run, [ product(queries), cursor ])