Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Limiting a queue over time

I'm using an API that is limited for usage, let's say: no more than 10 calls per second, and no more than 5000 calls per day.

I am handling this calls in a beanstalkd queue process job. How can I limit the processing of this jobs, having in mind the API's limits.

like image 587
catalinux Avatar asked Nov 01 '22 05:11

catalinux


1 Answers

When you use Beanstalkd you can have the tube paused for a certain seconds.

When you reserve a job, and you know the API call failed during that call, you get to pause the tube for X seconds.

You can find out the time needed to pause the tube, either from your API response (usually they return you are locked until Time X), or start with something adaptive like pause for the next 60 seconds, and increase/decrease on the go.

If you know you can delay, or disperse in advance, before placing the jobs into your queues, you can also add a delay to the job, so it won't execute immediately, this way you can have your jobs distributed over time.

Also there is a great post about distributed rate limiting using redis

like image 80
Pentium10 Avatar answered Nov 09 '22 13:11

Pentium10