Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Can a Celery task's definition and implementation be split?

Tags:

python

celery

I'm using Celery in Python to run background tasks and couldn't find any definitive answer to the question of whether I can split the Celery task definition from task implementation?

For example, take the really simple task below:

@celery_app.task
def add_numbers(num1, num2):
    return num1 + num2

The definition and implementation are in the same file i.e. when the caller imports this module to call add_numbers, both the definition and implementation are imported.

In this case, not so bad. But my tasks are a bit more complex, importing multiple modules and packages that the caller certainly doesn't need and I'd like to keep out of the caller.

So, does Celery provide a way to do this? Or am I going against the framework? Is this even a problem?

I have seen this question Celery dynamic tasks / hiding Celery implementation behind an interface implementation-behind-an-interface, but it is well over two years old - more than enough time for a lot to change.

like image 307
user783836 Avatar asked Oct 18 '14 15:10

user783836


People also ask

Does Celery support multiprocessing?

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operations but supports scheduling as well. The execution units, called tasks, are executed concurrently on one or more worker servers using multiprocessing, Eventlet, or gevent.

Is Celery multi threaded?

So Celery (and other queue frameworks) has other benefits as well - Think of it as a 'task/function manager' rather then just a way of multithreading.

How many tasks can Celery handle?

celery beats only trigger those 1000 tasks (by the crontab schedule), not run them. If you want to run 1000 tasks in parallel, you should have enough celery workers available to run those tasks.


1 Answers

There's a feature called signatures which allows calling tasks without importing them. You will need the Celery app instance to be available:

sig = celery_app.signature('myapp.add_numbers', args=(1,2))
sig.delay()
like image 196
tuomur Avatar answered Oct 05 '22 07:10

tuomur