Menu
I'm learning about python concurrency and I was introduced with the concept of futures. I read that as_completed() takes an iterable of futures and yields them as they are done.
I want to know how it works internally. Is it yielding completed tasks (futures) immediately? A naive approach would be to iterate all futures and examine each and every future using done(), but this is inefficient.
So what's the magic behind this function?
Thanks!
899
The concurrent. futures module provides a high-level interface for asynchronously executing callables. The asynchronous execution can be performed with threads, using ThreadPoolExecutor , or separate processes, using ProcessPoolExecutor .
You can get results from the ThreadPoolExecutor in the order that tasks are completed by calling the as_completed() module function. The function takes a collection of Future objects and will return the same Future objects in the order that their associated tasks are completed.
The ThreadPoolExecutor map() function supports target functions that take more than one argument by providing more than one iterable as arguments to the call to map(). For example, we can define a target function for map that takes two arguments, then provide two iterables to the call to map().
Many times the concurrent processes need to access the same data at the same time. Another solution, than using of explicit locks, is to use a data structure that supports concurrent access. For example, we can use the queue module, which provides thread-safe queues. We can also use multiprocessing.
I want to know how it works internally.
as_completed sets up a callback to fire when the future is done, doing so for all the futures it receives. (It uses an internal API equivalent to add_done_callback for this purpose.) When any of the futures completes, as_completed is notified by its callback being run. The callback runs in whatever thread it was that completed the future, so it only sets an event, which is shared by all callbacks, and which as_completed sleeps on. Once woken up by the event, as_completed immediately yields the finished future. This is how as_completed ensures that futures are yielded as they are completed, regardless of the order in which that happens. After yielding, the event is cleared and the waiting is repeated until all the futures are done.
Is it yielding completed tasks (futures) immediately?
Yes, that follows from both the documented interface and the implementation.
78
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
