Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Import celery task without importing dependencies

Tags:

python

celery

I have two modules

alpha.py
beta.py

beta.py can only be run on beta.server because it requires a licensed solver than only exists on beta.server.

Within alpha.py, there's a portion of code that calls:

beta_task.apply_async(kwargs={...})

As such, it requires

from beta import beta_task

Which in turn requires the magical proprietary module that is only available on beta.server.

I need to enable alpha_task to run on alpha.server, having the ability to call beta_task without having the beta_task code on the server.

Is this possible?

UPDATE

Also, can I prevent beta.task from running on alpha.server?

Since alpha.py import beta.py, the daemon finds beta.task and listens for tasks of this type:

- ** ---------- [config]
- ** ---------- .> app:         app_app
- ** ---------- .> transport:   asdfasdfasd
- ** ---------- .> results:     adfasdfasdf
- *** --- * --- .> concurrency: 12 (prefork)
-- ******* ----
--- ***** ----- [queues]
 -------------- .> celery           exchange=celery(direct) key=celery

 [tasks]
  . alpha.alpha_task
  . beta.beta_task
like image 540
artdv Avatar asked Nov 30 '22 23:11

artdv


1 Answers

I was able to get something like this working in my project by putting each module which needed to be separated in a different project and instantiating a Celery App in each with the same name. I then called the task by name.

# Import the app you created in the celeryconfig.py for your project
from celeryconfig import app

app.send_task('beta.tasks.beta_task', args=[2, 2], kwargs={})

I then can run one project on one server and the other on a different server and they are connected via the broker, but do not actually need to import code from each other.

like image 105
Kurt Wheeler Avatar answered Dec 02 '22 13:12

Kurt Wheeler