Moving Celery chain to a dead letter queue automatically when a task within fails

Question

I'm using Celery with RabbitMQ and I have a chain with 4 tasks and a group. The group is at the top of the chain and has around 1k tasks in it.

I have the dead letter queue setup and it works as expected with the individual tasks.

Whenever a task fails within the group, it goes to the dead letter queue, however the rest of the chain gets lost.

Chain:

1. group(a, b, c, d, e)
2. task x
3. task y
4. task w
5. task z

I want all of those tasks in the group to run in parallel, go to the dead letter queue if anything happens and move the chain task along with it (this could be after all the tasks run in the group). Is there any way of doing this? If not, what is the alternative way of achieving a recoverable chain with a group in it?

Answer 1

The chain itself is looking for a callback from the group, if the group fails the rest of the tasks will fail and be "lost"

Definition of a Chain: The chain primitive lets us link together signatures so that one is called after the other, essentially forming a chain of callbacks.

If I understand this correctly, you can handle this callback in the next tasks by defining the exception.

As a reference from a similar question: (Run a chord callback even if the main tasks fail)

Sources: https://docs.celeryproject.org/en/stable/userguide/canvas.html#chains

(I would have commented but I do not have enough reputation, hope this helps!)