Menu
I have some tasks in an application that are CPU bound and I want to use the multiprocessing module to use the multi-cores processors. I take a big task (a video file analysis) and I split it into several smaller tasks which are put in a queue and done by worker processes. What I want to know is how to report progress to the main process from these worker processes. For example I need them to send "I am at 1000ms of my analysis of file 1". What is the best way to do such progress reports ?
634
Using queues, tqdm-multiprocess supports multiple worker processes, each with multiple tqdm progress bars, displaying them cleanly through the main process. The worker processes also have access to a single global tqdm for aggregate progress monitoring.
Daemon processes in Python Python multiprocessing module allows us to have daemon processes through its daemonic option. Daemon processes or the processes that are running in the background follow similar concept as the daemon threads. To execute the process in the background, we need to set the daemonic flag to true.
Shared memory can be a very efficient way of handling data in a program that uses concurrency. Python's mmap uses shared memory to efficiently share large amounts of data between multiple Python processes, threads, and tasks that are happening concurrently.
imap_unordered instead of pool. imap will not have a large effect on the total running time of your code. It might be a little faster, but not by too much. What it may do, however, is make the interval between values being available in your iteration more even.
I would recommend a multiprocessing.Queue: nothing easier than for the worker processes to post their updates (presumably as tuples with the various aspect of their progress updates) there, while the main process just wait for such messages and when they come updates the GUI (or textual UI;-) to keep the user appraised of progress.
140
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
