progress bar slows down code by factor of 5 using tqdm and multiprocess

Question

I added a progress bar to my 2.7 python code using tqdm but it has slowed down my code significantly. Without the progress bar for one example it takes 12 seconds while with the progress bar it takes 57 seconds.

The code without the progress bar looks like this:

p = mp.Pool()
combs = various combinations
result = p.map(self.parallelize, combs)
p.close()
p.join()

The code with the progress bar is as follows:

from tqdm import tqdm
p = mp.Pool()
combs = various combinations
result = list(tqdm(p.imap(self.parallelize, combs), total = 5000))
p.close()
p.join()

Is there a better way that wouldn't slow down my code as much?

Answer 1

Can it be related to the usage of map and imap rather than twdm? See this great answer from the community. multiprocessing.Pool: What's the difference between map_async and imap?

Plus, you can adjust the update frequency of tqdm with the ministers parameter. If it is really related to tqdm reducing the update frequency might solve your problem.

miniters : int or float, optional Minimum progress display update interval, in iterations. If 0 and dynamic_miniters, will automatically adjust to equal mininterval (more CPU efficient, good for tight loops). If > 0, will skip display of specified number of iterations. Tweak this and mininterval to get very efficient loops. If your progress is erratic with both fast and slow iterations (network, skipping items, etc) you should set miniters=1.

https://github.com/tqdm/tqdm#usage