Menu
I have the following function:
def copy_file(source_file, target_dir): pass Now I would like to use multiprocessing to execute this function at once:
p = Pool(12) p.map(lambda x: copy_file(x,target_dir), file_list) The problem is, lambda's can't be pickled, so this fails. What is the most neat (pythonic) way to fix this?
317
To work with map(), the lambda should have one parameter in, representing one element from the source list. Choose a suitable name for the parameter, like n for a list of numbers, s for a list of strings. The result of map() is an "iterable" map object which mostly works like a list, but it does not print.
The map() function in Python takes in a function and a list as an argument. The function is called with a lambda function and a list and a new list is returned which contains all the lambda modified items returned by that function for each item.
Syntax. Simply put, a lambda function is just like any normal python function, except that it has no name when defining it, and it is contained in one line of code. A lambda function evaluates an expression for a given argument. You give the function a value (argument) and then provide the operation (expression).
The pool's map method chops the given iterable into a number of chunks which it submits to the process pool as separate tasks. The pool's map is a parallel equivalent of the built-in map method. The map blocks the main execution until all computations finish. The Pool can take the number of processes as a parameter.
Use a function object:
class Copier(object): def __init__(self, tgtdir): self.target_dir = tgtdir def __call__(self, src): copy_file(src, self.target_dir) To run your Pool.map:
p.map(Copier(target_dir), file_list) If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
