Using Python threads to make thousands of calls to a slow API with a rate limit

Question

I want to make thousands of calls to an API which is kind of slow -- tens of seconds to get a response. The only limit is that I can make at most one request per second. What's the best way to do this? I think the following code works, but I feel I should be able to make better use of the threading library somehow. I'm using python 3.3

last_job = datetime.now()
for work in work_list:
    while (datetime.now()-last_job).total_seconds() < 1 or threading.active_count() >= max_threads:
        time.sleep(.1)
    threading.Thread(target=work_function, args=[work]).start()
    last_job = datetime.now()

Answer 1

If you want to run a bunch of jobs using a fixed-size thread pool, you can use concurrent.futures.ThreadPoolExecutor, like this:

from concurrent.futures import ThreadPoolExecutor
with ThreadPoolExecutor(max_workers=5) as executor:
    for work in work_list:
        executor.submit(work_function, work)

If you want to ensure that you make at most one API call a second, then you need to do this from inside your work_function. You can't do it when submitting the job, because you don't know how long the job will queue up waiting for a thread to become available.

If it were me, I'd put the rate limiting code into its own class so that it's reusable:

from collections import Iterator
from threading import Lock
import time

class RateLimiter(Iterator):
    """Iterator that yields a value at most once every 'interval' seconds."""
    def __init__(self, interval):
        self.lock = Lock()
        self.interval = interval
        self.next_yield = 0

    def __next__(self):
        with self.lock:
            t = time.monotonic()
            if t < self.next_yield:
                time.sleep(self.next_yield - t)
                t = time.monotonic()
            self.next_yield = t + self.interval

api_rate_limiter = RateLimiter(1)

def work_function(work):
    next(api_rate_limiter)
    call_api(...)

time.monotonic was introduced in Python 3.3; in older versions of Python you could use time.time but this can jump backwards when the system clock changes, so you would need to ensure that this doesn't cause overlong sleeps:

                time.sleep(min(self.next_yield - t, self.interval))