ThreadPoolExecutor: how to limit the queue maxsize?

Question

I am using ThreadPoolExecutor class from the concurrent.futures package

def some_func(arg):     # does some heavy lifting     # outputs some results  from concurrent.futures import ThreadPoolExecutor  with ThreadPoolExecutor(max_workers=1) as executor:     for arg in range(10000000):         future = executor.submit(some_func, arg) 

but I need to limit the queue size somehow, as I don't want millions of futures to be created at once, is there a simple way to do it or should I stick to queue.Queue and threading package to accomplish this?

Answer 1

Python's ThreadPoolExecutor doesn't have the feature you're looking for, but the provided class can be easily sub-classed as follows to provide it:

from concurrent import futures import queue  class ThreadPoolExecutorWithQueueSizeLimit(futures.ThreadPoolExecutor):     def __init__(self, maxsize=50, *args, **kwargs):         super(ThreadPoolExecutorWithQueueSizeLimit, self).__init__(*args, **kwargs)         self._work_queue = queue.Queue(maxsize=maxsize) 

Answer 2

from concurrent.futures import ThreadPoolExecutor, wait, FIRST_COMPLETED  limit = 10  futures = set()  with ThreadPoolExecutor(max_workers=1) as executor:     for arg in range(10000000):         if len(futures) >= limit:             completed, futures = wait(futures, return_when=FIRST_COMPLETED)         futures.add(executor.submit(some_func, arg))