Menu
I'm trying to set up a daily task for my Django application on Elastic Beanstalk. There doesn't appear to be an accepted way to set this up, as celery beat is the go-to solution for periodic tasks in Django, but isn't great for load-balanced environments.
I've seen some solutions doing things like setting up celery beat with leader_only=True, to only run one instance, but that leaves a single point of failure. I've seen other solutions that allow many instances of celery beat and use locks to make sure only one task goes through, but wouldn't this still eventually fail completely unless the failed instances were restarted? Another suggestion I've seen is to have a separate instance for running celery beat, but this would still be a problem unless it had some way of restarting itself if it failed.
Are there any decent solutions to this problem? I would much rather not have to babysit a scheduler, as it would be pretty easy to not notice that my task was not being run until a while later.
913
To create periodic tasks, we need to define them using the beat_scheduler setting. Celery beat checks the beat_scheduler setting to manage the tasks that need to be executed periodically. To the purpose of my example I use Redis as message broker. So the first step is to tell Celery who is his messages broker.
celery-beat acts as the scheduler part of celery whereas the worker executes the tasks that are either instructed from within the application or by celery-beat . The installation steps for celery in a Django application is explained in celery docs here (after pip install celery 🙃).
We can configure periodic tasks either by manually adding the configurations to the celery.py module or using the django-celery-beat package which allows us to add periodic tasks from the Django Admin by extending the Admin functionality to allow scheduling tasks.
About. This extension enables you to store the periodic task schedule in the database. The periodic tasks can be managed from the Django Admin interface, where you can create, edit and delete periodic tasks and how often they should run.
If you're using redis as your broker, look into installing RedBeat as the celery beat scheduler: https://github.com/sibson/redbeat
This scheduler uses locking in redis to make sure only a single beat instance is running. With this you can enable beat on each node's worker process and remove the use of leader_only=True.
celery worker -B -S redbeat.RedBeatScheduler
Let's say you have Worker A with beat lock and Worker B. If Worker A dies, Worker B will attempt to acquire the beat lock after a configurable amount of time.
92
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
