I'm trying to get multiprocess.apply_async
to take in both *args
and **kwargs
. The docs indicate that this might be possible with the calling sequence:
apply_async(func[, args[, kwds[, callback]]])
But I can not figure out how to get the calling syntax correct. With the minimal example:
from multiprocessing import Pool
def f(x, *args, **kwargs):
print x, args, kwargs
args, kw = (), {}
print "# Normal call"
f(0, *args, **kw)
print "# Multicall"
P = Pool()
sol = [P.apply_async(f, (x,), *args, **kw) for x in range(2)]
P.close()
P.join()
for s in sol: s.get()
This works as expected giving the output
# Normal call
0 () {}
# Multicall
0 () {}
1 () {}
When args is not an empty tuple, for example args = (1,2,3)
, the single call works, but the multiprocessing solution gives:
# Normal call
0 (1, 2, 3) {}
# Multicall
Traceback (most recent call last):
File "kw.py", line 16, in <module>
sol = [P.apply_async(f, (x,), *args, **kw) for x in range(2)]
TypeError: apply_async() takes at most 5 arguments (6 given)
With the kwargs argument I get, for example kw = {'cat':'dog'}
# Normal call
0 () {'cat': 'dog'}
# Multicall
Traceback (most recent call last):
File "kw.py", line 15, in <module>
sol = [P.apply_async(f, (x,), *args, **kw) for x in range(2)]
TypeError: apply_async() got an unexpected keyword argument 'cat'
How do I properly wrap multiprocess.apply_async
?
You don't have to use *
and **
explicitly. Simply pass the tuple and the dict and let apply_async
unpack them:
from multiprocessing import Pool
def f(x, *args, **kwargs):
print x, args, kwargs
args, kw = (1,2,3), {'cat': 'dog'}
print "# Normal call"
f(0, *args, **kw)
print "# Multicall"
P = Pool()
sol = [P.apply_async(f, (x,) + args, kw) for x in range(2)]
P.close()
P.join()
for s in sol: s.get()
Output:
# Normal call
0 (1, 2, 3) {'cat': 'dog'}
# Multicall
0 (1, 2, 3) {'cat': 'dog'}
1 (1, 2, 3) {'cat': 'dog'}
Remember that in python's documentation, if a function accepts *args
and **kwargs
its signature explicitly states that:
the_function(a,b,c,d, *args, **kwargs)
In your case:
apply_async(func[, args[, kwds[, callback]]])
There are no *
there, hence args
is one argument, which is unpacked when calling func
and kwargs
is one argument and processed in the same way. Also note that it's not possible to have other arguments after **kwargs
:
>>> def test(**kwargs, something=True): pass
File "<stdin>", line 1
def test(**kwargs, something=True): pass
^
SyntaxError: invalid syntax
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With