Menu
I'm using Pool to multithread my programme using starmap to pass arguments.
I'm stuck because I cannot seem to find a way to pass kwargs along with the zip arrays that I'm passing in the starmap function.
pool = Pool(NO_OF_PROCESSES)
branches = pool.starmap(fetch_api, zip(repeat(project_name), api_extensions))
The branches request is incomplete as I'm still not able to figure out how to pass keywords arguments.
def fetch_api(project_name, api_extension, payload={}, headers={}, API_LINK=API_LINK, key=False):
headers[AUTH_STRING] = 'Gogo'
call_api = API_LINK + project_name + api_extension
response_api = requests.get(call_api, headers=headers, params=payload)
if key: return project_name + ':' + response_api
else: return response_api
While calling fetch_api() from the branch line, I want to pass payload as {'a':1} and key=True.
Please guide me to the direction or answer. Thanks. Using Python 3.3+.
951
Passing Keyword Arguments to Multiprocessing Processes We can also pass in arguments corresponding to the parameter name using the kwargs parameter in the Process class. Instead of passing a tuple, we pass a dictionary to kwargs where we specify the argument name and the variable being passed in as that argument.
map() function the Pool. starmap() allows us to issue tasks in chunks to the process pool. That is, we can group a fixed number of items from the input iterable and issue them as one task to be executed by a child worker process.
Using Pool. The Pool class in multiprocessing can handle an enormous number of processes. It allows you to run multiple jobs per process (due to its ability to queue the jobs). The memory is allocated only to the executing processes, unlike the Process class, which allocates memory to all the processes.
Python **kwargs allows function call to pass variable number of keyword (named) arguments to the function. The datatype of kwargs is dictionary. So, keywords and respective argument values come as key:value pairs. The number of key:value pairs in kwargs is determined only by the function call at the runtime.
Double asterisk ** before kwargs is the unpacking operator. It unpacks the arguments passed to the function, as dictionary. In this tutorial, we will learn how to use **kwargs in a function definition to accept any number of named arguments to the function. In this example, we will demonstrate a simple use case, just what **kwargs do.
Like the pool.map (function, iterable) method, the pool.starmap (function, iterable) method returns an iterator that applies the function provided as input to each item of the iterable. Still, it expects each input item iterable to be arranged as input function argument iterables.
You can use *args and **kwargs in a function definition to accept both positional arguments and named arguments, whose count is unknown. In the following example, we will define a function with both *args and **kwargs.
You can create a wrapper around pool.starmap that also accepts an iterator of over kwargs dictionaries.
from itertools import repeat
def starmap_with_kwargs(pool, fn, args_iter, kwargs_iter):
args_for_starmap = zip(repeat(fn), args_iter, kwargs_iter)
return pool.starmap(apply_args_and_kwargs, args_for_starmap)
def apply_args_and_kwargs(fn, args, kwargs):
return fn(*args, **kwargs)
Then you can call it in your case as:
args_iter = zip(repeat(project_name), api_extensions)
kwargs_iter = repeat(dict(payload={'a': 1}, key=True))
branches = starmap_with_kwargs(pool, fetch_api, args_iter, kwargs_iter)
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
