Menu
I need to build a system which handles two types of tasks. One type can create more tasks of itself or of the other type. There will be very few workers (2-3) and only one host. The most important requirement is that the system should handle restarts gracefully: i.e. on restart, tasks that were in progress should start from scratch and the workers should pick up tasks which were queued prior to restart. Looking at Celery it appears to be suitable for this use case. However, I have a couple of questions:
1) Is Celery able to handle restarts of the whole system as described? (Celery + workers + broker, everything restarts, complete power failure).
2) What is the best broker to use for this scenario? I would prefer SQLAlchemy + SQLite as it is more "self contained", but if RabbitMQ is the right choice, given the requirement to handle full restarts gracefully, then I can go with it.
950
Once you integrate Celery into your app, you can send time-intensive tasks to Celery's task queue. That way, your web app can continue to respond quickly to users while Celery completes expensive operations asynchronously in the background.
In the above example, the task will retry after a 5 second delay (via countdown ) and it allows for a maximum of 7 retry attempts (via max_retries ). Celery will stop retrying after 7 failed attempts and raise an exception.
To cancel an already executing task with Celery and Python, we can use the revoke function. to call revoke with the task_id of the task to stop. And we set terminate to True to terminate the task.
1) Is Celery able to handle restarts of the whole system as described? (Celery + workers + broker, everything restarts, complete power failure).
Yes, When you create your own queue on RabbitMQ ( Persistent queue ) even after the restart of your server the task will be resumed and will be carried out by Celery.
2) What is the best broker to use for this scenario? I would prefer SQLAlchemy + SQLite as it is more "self contained", but if RabbitMQ is the right choice, given the requirement to handle full restarts gracefully, then I can go with it.
USe RabbitMQ. we are using RabbitMQ + Celery in production. So i would suggest you to use RabbitMQ as a broker.
Celery + RabbitMQ Best Practice
Message Reliability
Notes :
132
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
