Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

What's the difference between Celery task and subtask?

Tags:

python

celery

If I understood the tutorial correctly, Celery subtask supports almost the same API as task, but has the additional advantage that it can be passed around to other functions or processes.

Clearly, if that was the case, Celery would have simply replaced tasks with subtasks instead of keeping both (e.g., the @app.task decorator would have converted a function to a subtask instead of to a task, etc.). So I must be misunderstanding something.

What can a task do that a subtask can't?

Celery API changed quite a bit; my question is specific to version 3.1 (currently, the latest).

Edit:

I know the docs say subtasks are intended to be called from other tasks. My question is what prevents Celery from getting rid of tasks completely and using subtasks everywhere? They seem to be strictly more flexible/powerful than tasks:

# tasks.py
from celery import Celery
app = Celery(backend='rpc://')

@app.task
def add(x, y):
    # just print out a log line for testing purposes
    print(x, y)

# client.py
from tasks import add
add_subtask = add.subtask()
# in this context, it seems the following two lines do the same thing
add.delay(2, 2)
add_subtask.delay(2, 2)
# when we need to pass argument to other tasks, we must use add_subtask
# so it seems add_subtask is strictly better than add
like image 954
max Avatar asked Oct 02 '16 19:10

max


People also ask

How does Celery task work?

Celery is a distributed task queue for UNIX systems. It allows you to offload work from your Python app. Once you integrate Celery into your app, you can send time-intensive tasks to Celery's task queue.

Why do we use sub tasks?

Subtasks are important because they help project managers accurately assign resources (such as project hours, personnel, and budget) to each task. They're also great for forecasting since they give a clearer picture of where there might be conflicts within the project itself.

Is Celery a task queue?

Celery is an open source asynchronous task queue or job queue which is based on distributed message passing. While it supports scheduling, its focus is on operations in real time.

Are celery tasks asynchronous?

Celery is a task queue/job queue based on asynchronous message passing. It can be used as a background task processor for your application in which you dump your tasks to execute in the background or at any given moment. It can be configured to execute your tasks synchronously or asynchronously.


1 Answers

You will take the difference into account when you start using complex workflows with celery.

A signature() wraps the arguments, keyword arguments, and execution options of a single task invocation in a way such that it can be passed to functions or even serialized and sent across the wire.

Signatures are often nicknamed “subtasks” because they describe a task to be called within a task.

Also:

subtask‘s are objects used to pass around the signature of a task invocation, (for example to send it over the network)

Task is just a function definition wrapped with decorator, but subtask is a task with parameters passed, but not yet started. You may transfer the subtask serialized over network or, more used, call it within a group/chain/chord.

like image 102
baldr Avatar answered Sep 17 '22 15:09

baldr