Menu
What are the factors to consider when deciding what to set max_workers to in ThreadPoolExecutor from concurrent.futures?
As long as you can expect Python 3.5+ to be available, is there any reason not to set max_workers to None which will then "default to the number of processors on the machine, multiplied by 5" as described in the docs here? https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor
700
ThreadPoolExecutor Methods : submit(fn, *args, **kwargs): It runs a callable or a method and returns a Future object representing the execution state of the method. map(fn, *iterables, timeout = None, chunksize = 1) : It maps the method and iterables together immediately and will raise an exception concurrent. futures.
The concurrent. futures module provides a high-level interface for asynchronously executing callables. The asynchronous execution can be performed with threads, using ThreadPoolExecutor , or separate processes, using ProcessPoolExecutor .
There is no maximum number of worker threads in the ThreadPoolExecutor. Nevertheless, your system will have an upper limit of the number of threads you can create based on how much main memory (RAM) you have available.
Like the built-in map() function, the ThreadPoolExecutor map() function returns an iterable over the results returned by the target function applied to the provided iterable of items.
I don't think this question can be so generically solved; it will depend on each case.
From this answer:
The more threads you use, the higher concurrency you'll achieve (up to a point), but the less CPU cycles you'll get (as there will be context switches). You have to instrument your application under typical workloads to see what works best for you. There is no universally optimal solution for this.
140
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
