How is rate_limit enforced in Celery?

Question

I'm running a Django website where I use Celery to implement preventive caching - that is, I calculate and cache results even before they are requested by the user.

However, one of my Celery tasks could, in some situation, be called a lot (I'd say sightly quicker than it completes on average, actually). I'd like to rate_limit it so that it doesn't consume a lot of resources when it's actually not that useful.

However, I'd like first to understand how Celery's celery.task.base.Task.rate_limit attribute is enforced. Are tasks refused? Are they delayed and executed later?

Thanks in advance!

Answer 1

Rate limited tasks are never dropped, they are queued internally in the worker so that they execute as soon as they are allowed to run.

The token bucket algorithm does not specify anything about dropping packets (it is an option, but Celery does not do that).