Celery discover tasks in files with other filenames

Question

I'm starting to have so many celery tasks that I want to break them down into smaller files instead of having them in a big tasks.py, but I don't manage to get celery to discover them.

Working structure:

proj/
    app/
        tasks.py

Structure I would like to work:

proj/
    app/
        tasks/
            __init__.py
            task1.py
            task2.py
            ...

But here celery does not find my tasks. I tried setting the CELERY_IMPORTS, but then I have to specify each file, and there is so many files. Same if I import each file contents into __init__.py. I would like all tasks to be automatically discovered, or at least all tasks in files in the tasks directory.

I could of course list all files in each tasks directory into CELERY_IMPORTS but that seems rather ugly.

Any ideas if this is possible in a nice way?

Answer 1

You could just define __all__ in your tasks/__init__.py file. Without additional celery settings changes.

For example:

# app/tasks/__init__.py

from .file_with_tasks_1 import task1, task2
from .file_with_tasks_2 import task3

__all__ = [
    'task1',
    'task2',
    'task3',
]

Tested on celery v4.4.6

Answer 2

The only reason that celery defaults to searching tasks.py is the default argument to autodiscover_tasks:

./loaders/base.py:def autodiscover_tasks(packages, related_name='tasks'):

If you use the configuration recommended by the docs you can just call autodiscover_tasks with non-default values for related_name for the different file names where you expect to have tasks. For example here is our celery.py:

from __future__ import absolute_import

import os

from celery import Celery

# set the default Django settings module for the 'celery' program.
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "settings")

from django.conf import settings

app = Celery('app')

# Using a string here means the worker will not have to
# pickle the object when using Windows.
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS, related_name='tasks2')