Celery: Chaining tasks with multiple arguments

Question

The celery documentation tells me that if multiple tasks are chained together, the result of the first task will be the first argument of the next. My problem is, I can't get it to work when I have a task that returns multiple results.

Example:

@task()
def get_comments(url):
    #get the comments and the submission and return them as 2 objects
    return comments, submission

@task
def render_template(threadComments, submission):
    #render the objects into a html file
    #does not return anything

Now, if I call them in a chain like (get_comments(url) | render_template()).apply_asnc() python will throw an TypeError: render_template() takes exactly 2 arguments (0 given).

I can see that the results are not unwrapped and applied to the arguments. If I only call get_comments, I can do:

result = get_comments(url)
arg1, arg2 = result

and get both results.

Answer 1

There are two mistakes here.

First, you don't have to call get_comments() and render_template(). Instead, you should use the .s() task method. Like:

( get_comments.s(url) | render_template.s()).apply_async()

In your case, you launch the function first, and then tries to join functions results to a chain.

Second, actually, you don't return "two results" from your first task. Instead, you return a tuple, containing both results, and this tuple is passed to the second task as the single object.

Therefore, you should rewrite your second task as

@task
def render_template(comments_and_submission):
   comments, submission = comments_and_submission

If you fix these, it should work.