RuntimeError: 'list' must be None or a list, not <class 'str'> while trying to start celery worker

Question

I am trying to add a celery task while following First Steps With Django but I get the following error:

Traceback (most recent call last):
  File "/Users/amrullahzunzunia/virtualenvs/flyrobe_new/bin/celery", line 11, in <module>
    sys.exit(main())
  File "/Users/amrullahzunzunia/virtualenvs/flyrobe_new/lib/python3.5/site-packages/celery/__main__.py", line 30, in main
    main()
  File "/Users/amrullahzunzunia/virtualenvs/flyrobe_new/lib/python3.5/site-packages/celery/bin/celery.py", line 81, in main
    cmd.execute_from_commandline(argv)
  File "/Users/amrullahzunzunia/virtualenvs/flyrobe_new/lib/python3.5/site-packages/celery/bin/celery.py", line 770, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/Users/amrullahzunzunia/virtualenvs/flyrobe_new/lib/python3.5/site-packages/celery/bin/base.py", line 311, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
  File "/Users/amrullahzunzunia/virtualenvs/flyrobe_new/lib/python3.5/site-packages/celery/bin/celery.py", line 762, in handle_argv
    return self.execute(command, argv)
  File "/Users/amrullahzunzunia/virtualenvs/flyrobe_new/lib/python3.5/site-packages/celery/bin/celery.py", line 694, in execute
    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
  File "/Users/amrullahzunzunia/virtualenvs/flyrobe_new/lib/python3.5/site-packages/celery/bin/worker.py", line 179, in run_from_argv
    return self(*args, **options)
  File "/Users/amrullahzunzunia/virtualenvs/flyrobe_new/lib/python3.5/site-packages/celery/bin/base.py", line 274, in __call__
    ret = self.run(*args, **kwargs)
  File "/Users/amrullahzunzunia/virtualenvs/flyrobe_new/lib/python3.5/site-packages/celery/bin/worker.py", line 212, in run
    state_db=self.node_format(state_db, hostname), **kwargs
  File "/Users/amrullahzunzunia/virtualenvs/flyrobe_new/lib/python3.5/site-packages/celery/worker/__init__.py", line 95, in __init__
    self.app.loader.init_worker()
  File "/Users/amrullahzunzunia/virtualenvs/flyrobe_new/lib/python3.5/site-packages/celery/loaders/base.py", line 128, in init_worker
    self.import_default_modules()
  File "/Users/amrullahzunzunia/virtualenvs/flyrobe_new/lib/python3.5/site-packages/celery/loaders/base.py", line 116, in import_default_modules
    signals.import_modules.send(sender=self.app)
  File "/Users/amrullahzunzunia/virtualenvs/flyrobe_new/lib/python3.5/site-packages/celery/utils/dispatch/signal.py", line 166, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/Users/amrullahzunzunia/virtualenvs/flyrobe_new/lib/python3.5/site-packages/amqp/utils.py", line 42, in __call__
    self.set_error_state(exc)
  File "/Users/amrullahzunzunia/virtualenvs/flyrobe_new/lib/python3.5/site-packages/amqp/utils.py", line 39, in __call__
    **dict(self.kwargs, **kwargs) if self.kwargs else kwargs
  File "/Users/amrullahzunzunia/virtualenvs/flyrobe_new/lib/python3.5/site-packages/celery/app/base.py", line 330, in _autodiscover_tasks
    self.loader.autodiscover_tasks(packages, related_name)
  File "/Users/amrullahzunzunia/virtualenvs/flyrobe_new/lib/python3.5/site-packages/celery/loaders/base.py", line 252, in autodiscover_tasks
    related_name) if mod)
  File "/Users/amrullahzunzunia/virtualenvs/flyrobe_new/lib/python3.5/site-packages/celery/loaders/base.py", line 273, in autodiscover_tasks
    return [find_related_module(pkg, related_name) for pkg in packages]
  File "/Users/amrullahzunzunia/virtualenvs/flyrobe_new/lib/python3.5/site-packages/celery/loaders/base.py", line 273, in <listcomp>
    return [find_related_module(pkg, related_name) for pkg in packages]
  File "/Users/amrullahzunzunia/virtualenvs/flyrobe_new/lib/python3.5/site-packages/celery/loaders/base.py", line 295, in find_related_module
    _imp.find_module(related_name, pkg_path)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/imp.py", line 270, in find_module
    "not {}".format(type(name)))
RuntimeError: 'list' must be None or a list, not <class 'str'>

This is my project structure:

• project
  • config
    • settings
      • base.py
      • local.py
      • production.py
    • celery.py # has a celery app
    • urls.py
    • wsgi.py
  • miscellaneous
    • models.py
    • views.py
    • tasks.py # has a celery task

This is my config/celery.py:

from __future__ import absolute_import
import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings.local')

from django.conf import settings

app = Celery('config')

app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)


@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))

This is my config/settings/base.py:

THIRD_PARTY_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'django.contrib.postgres',
    'django.contrib.gis',
    'oauth2_provider',
    'rest_framework',
    'rest_framework_gis',
    'import_export',
    'braces',
    'social.apps.django_app.default',
    'rest_framework_social_oauth2',
]

CUSTOM_APPS = [
    'miscellaneous',
    # more apps
]

INSTALLED_APPS = THIRD_PARTY_APPS + CUSTOM_APPS

BROKER_URL = 'redis://localhost:6379'
CELERY_RESULT_BACKEND = 'redis://localhost:6379'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'Asia/Kolkata'

This is my config/settings/local.py:

from .base import *
LOCAL_APPS = [
    'debug_toolbar',
]
INSTALLED_APPS.extend(LOCAL_APPS)

This is my miscellaneous/tasks.py:

from celery import shared_task

@shared_task
def log_request_meta(request_meta):
    # this is not complete yet
    return {"a": "b"}

I am using python-3.5 and django 1.9. I am unable to figure out what did I miss or where am I mistaken, because I did exactly what was in the tutorial mentioned above.

Update: Celery version is 3.1.20

Answer 1

This is how I solved the problem:

I saw that one of my apps was missing __init__.py, which caused problem with app.autodiscover_tasks(settings.INSTALLED_APPS)

I added the missing __init__.py and celery worker started without any issues

Answer 2

This error is raised by the Celery autodiscover_tasks when it cannot load one of your INSTALLED_APPS for some reason. In my case it was a directory with tasks.py file but other app files (models.py etc) missing because of an incomplete Git commit.

Answer 3

First problem - I believe the line should actually be

app.autodiscover_tasks(settings.INSTALLED_APPS)

Why did you use a lambda?

The second traceback - in miscellaneous/middleware.py you are invoking a celery task that passes a wsgiref.util.FileWrapper object - however you are using the JSON serializer which cannot serialize object instances - you'll need to use the pickle serializer instead.

See the relevant section in the docs.