Calling celery tasks requires all worker dependencies?

Question

Say I have a celery worker that depends on a large module, X.

Since task definitions require a reference to the worker app def (e.g., @app.task) this implies that my "client" (the code scheduling the task) needs also to depend on this module.

This doesn't make sense to me -- have I got this wrong?

A). I don't want my task caller to have these dependencies (e.g., they might be in different docker containers).

B). For security reasons I don't want my task caller to have access to this code.

Is there a way around it?

Thanks,

RB

Answer 1

Your client code can start tasks remotely without having to import the implementation of the tasks. You must obviously configure the client to connect to the same broker as the workers but once that is done, then you can use signatures to invoke the tasks:

import celery
result = celery.signature("app.tasks.foo", args=(1, )).delay().get()

The first parameter to celery.signature is the name of the task. It is typically the absolute name of the module that contains the task (e.g. app.tasks in the code above) plus the task name (foo).