Menu
I'm using Celery with RabbitMQ and I have a chain with 4 tasks and a group. The group is at the top of the chain and has around 1k tasks in it.
I have the dead letter queue setup and it works as expected with the individual tasks.
Whenever a task fails within the group, it goes to the dead letter queue, however the rest of the chain gets lost.
Chain:
I want all of those tasks in the group to run in parallel, go to the dead letter queue if anything happens and move the chain task along with it (this could be after all the tasks run in the group). Is there any way of doing this? If not, what is the alternative way of achieving a recoverable chain with a group in it?
249
By default, Celery routes all tasks to a single queue and all workers consume this default queue. With Celery queues, you can control which Celery workers process which tasks. This can be useful if you have a slow and a fast task and you want the slow tasks not to interfere with the fast tasks.
Task Routing in Celery 1 Step 1: Configure Celery via task_routes#N#Celery can be configured on a per-task basis which queue a task gets sent... 2 Step 2: Make worker subscribe to a queue#N#We run one Celery worker that subscribes to the feeds queue and processes... 3 Step 3: Give it a go More ...
There are generally two things that can go wrong as you send a task to a Celery worker to process it in the background. Connection issues with the broker and Message Queue. Exceptions raised on the worker.
A message is an information on what task to be executed and input parameters for the task. Cooking is a task to be executed in Celery. A task (in programming is a function) and contains the action/code which acts on an input and produces some output.
The chain itself is looking for a callback from the group, if the group fails the rest of the tasks will fail and be "lost"
Definition of a Chain: The chain primitive lets us link together signatures so that one is called after the other, essentially forming a chain of callbacks.
If I understand this correctly, you can handle this callback in the next tasks by defining the exception.
As a reference from a similar question: (Run a chord callback even if the main tasks fail)
Sources: https://docs.celeryproject.org/en/stable/userguide/canvas.html#chains
(I would have commented but I do not have enough reputation, hope this helps!)
79
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
