How to structure celery tasks

Question

I have 2 types of task: async tasks and schedule tasks. So, here is my dir structure:

proj   |   -- tasks       |       -- __init__.py       |       -- celeryapp.py     => celery instance defined in this file.       |       -- celeryconfig.py       |       -- async       |    |       |    -- __init__.py       |    |       |    -- task1.py    => from proj.tasks.celeryapp import celery       |    |       |    -- task2.py    => from proj.tasks.celeryapp import celery       |       -- schedule            |            -- __init__.py            |            -- task1.py    => from proj.tasks.celeryapp import celery            |            -- task2.py    => from proj.tasks.celeryapp import celery 

But when I run celery worker like below, it does not work. It can not accept the task from celery beat scheduler.

 $ celery worker --app=tasks -Q my_queue,default_queue 

So, is there any best practice on multiple task files organization?

Answer 1

Based on celery documentation you can import a structure of celery tasks like this:

For example if you have an (imagined) directory tree like this:

| |-- foo |    |-- __init__.py |    |-- tasks.py | |-- bar      |-- __init__.py      |-- tasks.py 

Then calling app.autodiscover_tasks(['foo', bar']) will result in the modules foo.tasks and bar.tasks being imported.