Detecting when a celery task and all subtasks have completed

Question

I have a parent task that will spawn an arbitrary and potentially largish number of subtasks. Once both the parent and all of the subtasks have completed I need to set a flag in my database to indicate that it's ready. How would I best go about doing that?

For example:

@task()
def master_task(foo):
    foo_obj = Foo.objects.get(id=foo)
    for bar in foo_obj.bar_set.all():
        more_work.delay(bar.id)

@task()
def more_work(bar):
   bar_obj = Bar.objects.get(id=bar)
   do_work()

I need to detect when the master_task and all of the subtasks it has spawns have completed so that I can set a flag on a related model to indicate that everything is ready

Answer 1

Use chords

You should use a [TaskSet][1]: > The TaskSet enables easy invocation of several tasks at once, and is then able to join the results in the same order as the tasks were invoked.

Answer 2

celery.chord is designed precisely for that.