I am currently doing a merge over a set of variables that I'd like to parallelize. My code looks something like this:
mergelist = [
('leftfile1', 'rightfile1', 'leftvarname1', 'outputname1'),
('leftfile1', 'rightfile1', 'leftvarname2', 'outputname2')
('leftfile2', 'rightfile2', 'leftvarname3', 'outputname3')
]
def merger(leftfile,rightfile,leftvarname,outvarname):
do_the_merge
for m in mergelist:
merger(*m)
Ordinarily, to speed up long loops, I would replace the for m in mergelist with something like....
from multiprocessing import Pool
p = Pool(8)
p.map(merger(m), mergelist)
p.close()
But since I'm using the star to unpack the tuple, it's not clear to me how to map this correctly. How do I get the *m
?
Use lambda
:
with Pool(8) as p:
p.map(lambda m:merger(*m), mergelist)
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With