Menu
I've implemented some simple parallelism in a Monte Carlo code using the Python multiprocessing module. I have code that looks like:
montecarlos = [MonteCarlo(f,fargs) for fargs in farglist] jobs = [multiprocessing.Process(mc) for mc in montecarlos] for job in jobs: job.start() for job in jobs: job.join() results = [mc.results for mc in montecarlos] However, when I look at the results list, it looks like the monte carlo iterators haven't even started. I know that they have, because I can have the processes print out information during the monte carlo steps. So I'm doing something dumb. I had thought the job.join() would keep the results list from being constructed until everything had run, and thus the mc.results field would be updated.
I realize I haven't told you the details of my monte carlo routine, and hope that it doesn't matter, and that the mistake I'm making is in my interpretation of what multiprocessing does. Thanks in advance for any help you can offer.
566
Python provides a number of process-safe queues, such as the multiprocessing. Queue class.
While the Process keeps all the processes in the memory, the Pool keeps only those that are under execution. Therefore, if you have a large number of tasks, and if they have more data and take a lot of space too, then using process class might waste a lot of memory. The overhead of creating a Pool is more.
This pattern is extremely common, and I illustrate it here with a toy stream processing application. On a machine with 48 physical cores, Ray is 6x faster than Python multiprocessing and 17x faster than single-threaded Python. Python multiprocessing doesn't outperform single-threaded Python on fewer than 24 cores.
Understand multiprocessing in no more than 6 minutes Multiprocessing is quintessential when a long-running process has to be speeded up or multiple processes have to execute parallelly. Executing a process on a single core confines its capability, which could otherwise spread its tentacles across multiple cores.
The MonteCarlo objects have been pickled and sent to child processes to be run - the .results attribute in this process isn't populated because the local mc has never been run.
If you create a multiprocessing.Queue, you can pass that into each MonteCarlo job, and when it finishes it should put the result in there. Then the top-level can wait for values from the queue. (Under the hood this will pickle and unpickle the result object.)
result_queue = multiprocessing.Queue() montecarlos = [MonteCarlo(result_queue, f,fargs) for fargs in farglist] jobs = [multiprocessing.Process(mc) for mc in montecarlos] for job in jobs: job.start() for job in jobs: job.join() results = [result_queue.get() for mc in montecarlos] If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
