Menu
I'm writing an app in Python and I need to run some tasks simultaneously. The module multiprocessing offers the class Process and the concurrent.futures module has the class ProcessPoolExecutor. Both seem to use multiple processes to execute their tasks, but their APIs are different. Why should I use one over the other?
I know that concurrent.futures was added in Python 3, so I guess it's better?
786
This provides us with a simple model for parallel execution on a multi-core machine. While concurrent futures provide a simpler interface, it is slower and less flexible when compared with using multiprocessing for parallel execution.
The concurrent. futures module provides a high-level interface for asynchronously executing callables. The asynchronous execution can be performed with threads, using ThreadPoolExecutor , or separate processes, using ProcessPoolExecutor .
By formal definition, multithreading refers to the ability of a processor to execute multiple threads concurrently, where each thread runs a process. Whereas multiprocessing refers to the ability of a system to run multiple processors concurrently, where each processor can run one or more threads.
All threads can share same set of open files, child processes. If one process is blocked, then no other process can execute until the first process is unblocked. While one thread is blocked and waiting, a second thread in the same task can run. Multiple processes without using threads use more resources.
The motivations for concurrent.futures are covered in the PEP.
In my practical experience concurrent.futures provides a more convenient programming model for long-running task submission and monitoring situations. A program I recently wrote using concurrent.futures involved monitoring a directory for incoming files over a 2-3 hour window, translating each file as it arrives to a task, submitting it and so on. Future objects returned by the ProcessPoolExecutor allow for tracking task status, providing intermediate status reports etc in a convenient way.
141
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
