I was finally getting to the point where I had some free time and wanted to add Celery to my Python/Flask project on Heroku. However, almost all mentions of Celery from the Heroku docs are gone. There used to be article with a tutotial in the "Getting started with Django", but it's gone.
Will "just doing it" myself work? What's a good AMQP addon to use as backend on Heroku?
Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well. What is Redis? An in-memory database that persists on disk. Redis is an open source, BSD licensed, advanced key-value store.
I think there are issues with Celery as a background task on Heroku. We tried to create such tasks and they take all memory after running for about 20 minutes, even with DEBUG=False on Redis or RabbitMQ. Worse still, the memory is NEVER released: every time we have to restart the worker. The same code runs flawlessly on bare Linux or on Mac with Foreman. It happens with very simple tasks, like reading a text file in a loop, writing to a Django model.
I'm the owner of Python at Heroku.
Celery is still fully supported. The documentation at the bottom of the Django guide was removed because it was out of date with the latest release of Celery (v3.0) and was fairly incomplete, in my opinion.
I'm currently working on a new Dev Center article that goes in depth on how to get the most out of Celery. It should be posted soon.
As for the queue addon — I'm a big fan of CloudAMPQ.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With