Django, ImportError: cannot import name Celery, possible circular import?

Question

I went through this example here:

http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html

All my tasks are in files called tasks.py.

After updating celery and adding the file from the example django is throwing the following error, no matter what I try:

ImportError: cannot import name Celery 

Is the problem possibly caused by the following?

app.autodiscover_tasks(settings.INSTALLED_APPS, related_name='tasks') 

Because it goes through all tasks.py files which all have the following import.

from cloud.celery import app 

cloud/celery.py:

from __future__ import absolute_import  import os, sys from celery import Celery from celery.schedules import crontab from django.conf import settings  BROKER_URL = 'redis://:PASSWORD@localhost'  os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'cloud.settings') app = Celery('cloud', broker=BROKER_URL) app.config_from_object('django.conf:settings') app.autodiscover_tasks(settings.INSTALLED_APPS, related_name='tasks')  if "test" in sys.argv:     app.conf.update(         CELERY_ALWAYS_EAGER=True,     )     print >> sys.stderr, 'CELERY_ALWAYS_EAGER = True'  CELERYBEAT_SCHEDULE = {     'test_rabbit_running': {         "task": "retail.tasks.test_rabbit_running",         "schedule": 3600, #every hour     },     [..]  app.conf.update(     CELERYBEAT_SCHEDULE=CELERYBEAT_SCHEDULE ) 

retail/tasks.py:

from cloud.celery import app  import logging from celery.utils.log import get_task_logger logger = get_task_logger('tasks') logger.setLevel(logging.DEBUG)  @app.task def test_rabbit_running():     import datetime     utcnow = datetime.datetime.now()     logger.info('CELERY RUNNING') 

The error happens, when I try to access a url that is not valid, like /foobar.

Here is the full traceback:

Traceback (most recent call last):   File "/opt/virtenvs/django_slice/local/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 126, in handle_request     respiter = self.wsgi(environ, resp.start_response)   File "/opt/virtenvs/django_slice/local/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 255, in __call__     response = self.get_response(request)   File "/opt/virtenvs/django_slice/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 178, in get_response     response = self.handle_uncaught_exception(request, resolver, sys.exc_info())   File "/opt/virtenvs/django_slice/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 220, in handle_uncaught_exception     if resolver.urlconf_module is None:   File "/opt/virtenvs/django_slice/local/lib/python2.7/site-packages/django/core/urlresolvers.py", line 342, in urlconf_module     self._urlconf_module = import_module(self.urlconf_name)   File "/opt/virtenvs/django_slice/local/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module     __import__(name)   File "/opt/src/slicephone/cloud/cloud/urls.py", line 52, in      urlpatterns +=  patterns('', url(r'^search/', include('search.urls')))   File "/opt/virtenvs/django_slice/local/lib/python2.7/site-packages/django/conf/urls/__init__.py", line 25, in include     urlconf_module = import_module(urlconf_module)   File "/opt/virtenvs/django_slice/local/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module     __import__(name)   File "/opt/src/slicephone/cloud/search/urls.py", line 5, in      from handlers import SearchHandler   File "/opt/src/slicephone/cloud/search/handlers.py", line 15, in      from places import handlers as placeshandler   File "/opt/src/slicephone/cloud/places/handlers.py", line 23, in      import api as placesapi   File "/opt/src/slicephone/cloud/places/api.py", line 9, in      from djapi import *   File "/opt/src/slicephone/cloud/places/djapi.py", line 26, in      from tasks import add_single_place, add_multiple_places   File "/opt/src/slicephone/cloud/places/tasks.py", line 2, in      from cloud.celery import app   File "/opt/src/slicephone/cloud/cloud/celery.py", line 4, in      from celery import Celery   File "/opt/src/slicephone/cloud/cloud/celery.py", line 4, in      from celery import Celery ImportError: cannot import name Celery 

Answer 1

Adding the following lines to cloud/celery.py:

import celery print celery.__file__ 

gave me the file itself and not the celery module from the library. After renaming celery.py to celeryapp.py and adjusting the imports all errors were gone.

Note:

That leads to a change in starting the worker:

celery worker --app=cloud.celeryapp:app 

For those running celery==3.1.2 and getting this error:

TypeError: unpack_from() argument 1 must be string or read-only buffer, not memoryview 

Apply the patch mentioned here: https://github.com/celery/celery/issues/1637

Answer 2

With Django 1.7.5, Celery 3.1.17, and Python 2.7.6 I found that I was still getting these ImportError: cannot import name Celery. But only when running tests under PyCharm 4.0.4.

I found that a solution was not to rely on from __future__ import absolute_import as described in First Steps with Django. Instead I renamed proj/proj/celery.py to proj/proj/celery_tasks.py and then changed the content of __init__.py to match: from .celery_tasks import app as celery_app. No more multiple instances of files named celery.py to cause import confusion seemed to be a simpler approach.