Menu
I am using ThreadPoolExecutor class from the concurrent.futures package
def some_func(arg): # does some heavy lifting # outputs some results from concurrent.futures import ThreadPoolExecutor with ThreadPoolExecutor(max_workers=1) as executor: for arg in range(10000000): future = executor.submit(some_func, arg) but I need to limit the queue size somehow, as I don't want millions of futures to be created at once, is there a simple way to do it or should I stick to queue.Queue and threading package to accomplish this?
970
The ProcessPoolExecutor in Python provides a process pool that lets you run tasks concurrently. You can add tasks to the pool by calling submit() with your function name, which will return a Future object. You can call the cancel() function on the Future object to cancel the task before it has started running.
The ThreadPoolExecutor is designed to speed-up your program by executing tasks concurrently. Nevertheless, in some use cases, using the ThreadPoolExecutor can make your program slower. Sometimes dramatically slower than performing the same task in a for loop.
The ThreadPoolExecutor Python class is used to create and manage thread pools and is provided in the concurrent. futures module.
ThreadPoolExecutor Thread-Safety Although the ThreadPoolExecutor uses threads internally, you do not need to work with threads directly in order to execute tasks and get results. Nevertheless, when accessing resources or critical sections, thread-safety may be a concern.
Python's ThreadPoolExecutor doesn't have the feature you're looking for, but the provided class can be easily sub-classed as follows to provide it:
from concurrent import futures import queue class ThreadPoolExecutorWithQueueSizeLimit(futures.ThreadPoolExecutor): def __init__(self, maxsize=50, *args, **kwargs): super(ThreadPoolExecutorWithQueueSizeLimit, self).__init__(*args, **kwargs) self._work_queue = queue.Queue(maxsize=maxsize) 
from concurrent.futures import ThreadPoolExecutor, wait, FIRST_COMPLETED limit = 10 futures = set() with ThreadPoolExecutor(max_workers=1) as executor: for arg in range(10000000): if len(futures) >= limit: completed, futures = wait(futures, return_when=FIRST_COMPLETED) futures.add(executor.submit(some_func, arg)) If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With