GAE: relationship between queue rate and max_concurrent_requests

Question

I'm reading the docs about GAE task queues and I'm unclear about the relationship (if any) between rate and max_concurrent_requests. Does rate refer to a single request so that e.g. rate=20/s and max_concurrent_requests=10 effectively means max 200 requests/s in total?

Answer 1

There are 3 parameters :

• bucket size ~ the maximum burst rate
• rate ~ normal processing rate
• max concurrent requests ~ additional concurrency restriction

When a tasks starts executing, it looks at the bucket to check if there is a token or slot free.

• If so, the number of tokens is decreased with one and the task starts executing if there are less than max concurrent requests already executing.
• if not, the task will have to wait for a new token/slot to arrive at the speed of rate.

If you have a big bucket, it may fill up when traffic is low and e.g. with a bucket size of 20 and a sudden peak in demand, you can start 20 tasks because they all can get a token. Then the bucket is empty and the tasks will be processed at rate speed.

Max concurrent requests can additionally limit the burst rate since long running tasks will slow down the whole processing. Without max concurrent requests, only the bucket size and rate determine the processing speed, not the duration of your tasks.