Limit method calls per second(s) (refuse when limit reached)

Question

I have a method that does some IO, and I'd like to limit the calls (per second) to this method to avoid the backend to get bursts of concurrent requests it can't handle.

If the requirement came without the "per second" I could just use a stack (basically just a counter) and offer() when starting a request, and poll() when done. With the "per second" requirement I'd somehow need to clean slots on the stack that are older than a given lapse of time.

How do I do that correctly? The structure should be thread-safe, obviously.

Thank you for your time!

Answer 1

Have a look at this question. The answer is equally applicable to this problem, even if the question is phrased rather differently.

I confess, I find it slightly mysterious, but it works beautifully.

Answer 2

You could use a queue but with some modifications. Whenever you want to add to the queue (the method gets called) you check how many elements are in it and when they were added. All elements that were inserted more than one second ago can be removed. Compare the number of remaining elements with the rate per second and you have a decision whether to reject the method execution or not. It might get a little bit more complicated if you intend to have the method block instead of reject.

There are likely to be many possible implementations and you have to check whether they are feasible for your problem. You could also take a look at implementations of Token buckets which is the general concept of your problem.