Pass multiple parameters to concurrent.futures.Executor.map?

Question

The concurrent.futures.Executor.map takes a variable number of iterables from which the function given is called. How should I call it if I have a generator that produces tuples that are normally unpacked in place?

The following doesn't work because each of the generated tuples is given as a different argument to map:

args = ((a, b) for (a, b) in c) for result in executor.map(f, *args):     pass 

Without the generator, the desired arguments to map might look like this:

executor.map(     f,     (i[0] for i in args),     (i[1] for i in args),     ...,     (i[N] for i in args), ) 

Answer 1

One argument that is repeated, one argument in c

from itertools import repeat for result in executor.map(f, repeat(a), c):     pass 

Need to unpack items of c, and can unpack c

from itertools import izip for result in executor.map(f, *izip(*c)):     pass 

Need to unpack items of c, can't unpack c

1. Change f to take a single argument and unpack the argument in the function.

2. If each item in c has a variable number of members, or you're calling f only a few times:

   executor.map(lambda args, f=f: f(*args), c) 

   It defines a new function that unpacks each item from c and calls f. Using a default argument for f in the lambda makes f local inside the lambda and so reduces lookup time.

3. If you've got a fixed number of arguments, and you need to call f a lot of times:

   from collections import deque def itemtee(iterable, n=2):     def gen(it = iter(iterable), items = deque(), next = next):         popleft = items.popleft         extend = items.extend         while True:             if not items:                 extend(next(it))             yield popleft()     return [gen()] * n  executor.map(f, *itemtee(c, n)) 

Where n is the number of arguments to f. This is adapted from itertools.tee.

Answer 2

You need to remove the * on the map call:

args = ((a, b) for b in c) for result in executor.map(f, args):     pass 

This will call f, len(args) times, where f should accept one parameter.

If you want f to accept two parameters you can use a lambda call like:

args = ((a, b) for b in c) for result in executor.map(lambda p: f(*p), args):   # (*p) does the unpacking part     pass