Limiting a queue over time

Question

I'm using an API that is limited for usage, let's say: no more than 10 calls per second, and no more than 5000 calls per day.

I am handling this calls in a beanstalkd queue process job. How can I limit the processing of this jobs, having in mind the API's limits.

Answer 1

When you use Beanstalkd you can have the tube paused for a certain seconds.

When you reserve a job, and you know the API call failed during that call, you get to pause the tube for X seconds.

You can find out the time needed to pause the tube, either from your API response (usually they return you are locked until Time X), or start with something adaptive like pause for the next 60 seconds, and increase/decrease on the go.

If you know you can delay, or disperse in advance, before placing the jobs into your queues, you can also add a delay to the job, so it won't execute immediately, this way you can have your jobs distributed over time.

Also there is a great post about distributed rate limiting using redis