How to resolve celery.backends.rpc.BacklogLimitExceeded error

Question

I am using Celery with Flask after working for a good long while, my celery is showing a celery.backends.rpc.BacklogLimitExceeded error. My config values are below:

CELERY_BROKER_URL = 'amqp://'
CELERY_TRACK_STARTED = True
CELERY_RESULT_BACKEND = 'rpc'
CELERY_RESULT_PERSISTENT = False

Can anyone explain why the error is appearing and how to resolve it? I have checked the docs here which doesnt provide any resolution for the issue.

Answer 1

Possibly because your process consuming the results is not keeping up with the process that is producing the results? This can result in a large number of unprocessed results building up - this is the "backlog". When the size of the backlog exceeds an arbitrary limit, BacklogLimitExceeded is raised by celery.

You could try adding more consumers to process the results? Or set a shorter value for the result_expires setting?

The discussion on this closed celery issue may help:

Seems like the database backends would be a much better fit for this purpose. The amqp/RPC result backends needs to send one message per state update, while for the database based backends (redis, sqla, django, mongodb, cache, etc) every new state update will overwrite the old one.

The "amqp" result backend is not recommended at all since it creates one queue per task, which is required to mimic the database based backends where multiple processes can retrieve the result.

The RPC result backend is preferred for RPC-style calls where only the process that initiated the task can retrieve the result.

But if you want persistent multi-consumer result you should store them in a database.

Using rabbitmq as a broker and redis for results is a great combination, but using an SQL database for results works well too.