django database as a celery broker giving connection refused error

Question

I am trying to follow instruction on http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html to make celery work with django.

As for broker I am trying to setup django database by following instruction on http://docs.celeryproject.org/en/latest/getting-started/brokers/django.html#broker-django.

I can successfully start a worker as you can see below:

*****-MacBook-Air:celeryproject *****$ python manage.py celery worker -E --loglevel=info /Library/Python/2.7/site-packages/django_celery-3.0.9-py2.7.egg/djcelery/loaders.py:124:
UserWarning: Using settings.DEBUG leads to a memory leak, never use
this setting in production environments!   warnings.warn("Using
settings.DEBUG leads to a memory leak, never "    --------------
celery@*****-MacBook-Air.local v3.0.9 (Chiastic Slide)  
---- **** -----   
--- * ***  * -- [Configuration]  
-- * - **** --- . broker:      django://localhost//  
- ** ---------- . app:         default:0x10b785fd0 (djcelery.loaders.DjangoLoader)  
- ** ---------- . concurrency: 4 (processes)  
- ** ---------- . events:      ON  
- ** ----------   
- *** --- * --- [Queues]   
-- ******* ---- . celery:      exchange:celery(direct) binding:celery  
--- ***** -----   

[Tasks]     . celeryapp.tasks.add  

[2012-09-06 20:03:23,711: WARNING/MainProcess] celery@*****-MacBook-Air.local has started.
[2012-09-06 20:03:24,406: WARNING/PoolWorker-3] /Library/Python/2.7/site- 
packages/django_celery-3.0.9-py2.7.egg/djcelery/loaders.py:124: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
warnings.warn("Using settings.DEBUG leads to a memory leak, never "
[2012-09-06 20:03:24,406: WARNING/PoolWorker-1] /Library/Python/2.7/site-packages/django_celery-3.0.9-py2.7.egg/djcelery/loaders.py:124: UserWarning: Using
settings.DEBUG leads to a memory leak, never use this setting in production environments!
warnings.warn("Using settings.DEBUG leads to a memory leak, never "
[2012-09-06 20:03:24,406: WARNING/PoolWorker-2] /Library/Python/2.7/site-packages/django_celery-3.0.9-py2.7.egg/djcelery/loaders.py:124: UserWarning: Using
settings.DEBUG leads to a memory leak, never use this setting in production environments!
warnings.warn("Using settings.DEBUG leads to a memory leak, never "

[2012-09-06 20:03:24,406: WARNING/PoolWorker-4] /Library/Python/2.7/site- packages/django_celery-3.0.9-py2.7.egg/djcelery/loaders.py:124: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments! warnings.warn("Using settings.DEBUG leads to a memory leak, never "

But when I continue following the instruction and call the add task created in the getting started steps, I run into following error:

Python 2.7.1 (r271:86832, Jul 31 2011, 19:30:53)   
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin  
Type "help", "copyright", "credits" or "license" for more information.  
>>> from celeryapp.tasks import add  
>>> add.delay(2,2)  
Traceback (most recent call last):  
  File "<stdin>", line 1, in <module>  
  File "/Library/Python/2.7/site-packages/celery-3.0.9-py2.7.egg/celery/app/task.py", line   343, in delay  
    return self.apply_async(args, kwargs)  
  File "/Library/Python/2.7/site-packages/celery-3.0.9-py2.7.egg/celery/app/task.py", line   458, in apply_async  
    with app.producer_or_acquire(producer) as P:  
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17, in __enter__  
    return self.gen.next()  
  File "/Library/Python/2.7/site-packages/celery-3.0.9-py2.7.egg/celery/app/base.py", line 256, in producer_or_acquire  
    with self.amqp.producer_pool.acquire(block=True) as producer:  
  File "/Library/Python/2.7/site-packages/kombu-2.4.5-py2.7.egg/kombu/connection.py", line  712, in acquire
    R = self.prepare(R)  
  File "/Library/Python/2.7/site-packages/kombu-2.4.5-py2.7.egg/kombu/pools.py", line 54, in prepare  
    p = p()  
  File "/Library/Python/2.7/site-packages/kombu-2.4.5-py2.7.egg/kombu/pools.py", line 45, in <lambda>  
    return lambda: self.create_producer()  
  File "/Library/Python/2.7/site-packages/kombu-2.4.5-py2.7.egg/kombu/pools.py", line 42, in create_producer  
    return self.Producer(self._acquire_connection())  
  File "/Library/Python/2.7/site-packages/celery-3.0.9-py2.7.egg/celery/app/amqp.py", line 160, in __init__  
    super(TaskProducer, self).__init__(channel, exchange, *args, **kwargs)  
  File "/Library/Python/2.7/site-packages/kombu-2.4.5-py2.7.egg/kombu/messaging.py", line 83, in __init__  
    self.revive(self.channel)  
  File "/Library/Python/2.7/site-packages/kombu-2.4.5-py2.7.egg/kombu/messaging.py", line 174, in revive
    channel = self.channel = maybe_channel(channel)  
  File "/Library/Python/2.7/site-packages/kombu-2.4.5-py2.7.egg/kombu/connection.py", line 886, in maybe_channel  
    return channel.default_channel  
  File "/Library/Python/2.7/site-packages/kombu-2.4.5-py2.7.egg/kombu/connection.py", line 624, in default_channel  
    self.connection  
  File "/Library/Python/2.7/site-packages/kombu-2.4.5-py2.7.egg/kombu/connection.py", line 617, in connection  
    self._connection = self._establish_connection()  
  File "/Library/Python/2.7/site-packages/kombu-2.4.5-py2.7.egg/kombu/connection.py", line 576, in _establish_connection  
    conn = self.transport.establish_connection()  
  File "/Library/Python/2.7/site-packages/kombu-2.4.5-py2.7.egg/kombu/transport/amqplib.py", line 344, in establish_connection  
    connect_timeout=conninfo.connect_timeout)  
  File "/Library/Python/2.7/site-packages/kombu-2.4.5-py2.7.egg/kombu/transport/amqplib.py", line 154, in __init__  
    super(Connection, self).__init__(*args, **kwargs)  
  File "build/bdist.macosx-10.7-intel/egg/amqplib/client_0_8/connection.py", line 129, in __init__  
  File "build/bdist.macosx-10.7-intel/egg/amqplib/client_0_8/transport.py", line 281, in create_transport  
  File "build/bdist.macosx-10.7-intel/egg/amqplib/client_0_8/transport.py", line 85, in __init__  
socket.error: [Errno 61] Connection refused  

Basically it seems that this app is unable to setup a transport with my broker.

Following is snippet of my settings.py:

# Django settings for celeryproject project.
import djcelery
djcelery.setup_loader()

## Broker settings.
BROKER_URL= 'django://'
BROKER_BACKEND = "djkombu.transport.DatabaseTransport"

# List of modules to import when celery starts.
CELERY_IMPORTS = ("celeryapp.tasks", )

## Using the database to store task state and results.
#CELERY_RESULT_BACKEND = "database"
#CELERY_RESULT_DBURI = "sqlite:///mydatabase.db"

CELERY_ANNOTATIONS = {"tasks.add": {"rate_limit": "10/s"}}

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'celeryapp',
    'djcelery',
    #'djcelery.transport',
    'kombu.transport.django',
    # Uncomment the next line to enable the admin:
    # 'django.contrib.admin',
    # Uncomment the next line to enable admin documentation:
    # 'django.contrib.admindocs',
)

Any idea?

Answer 1

try to add

import djcelery
djcelery.setup_loader()

after your broker and backend configuration, also according to https://devcenter.heroku.com/articles/django#running-a-worker guide, mb you need to change INSTALLED_APPS order

'kombu.transport.django',
'djcelery'

Answer 2

The issue seems to be that the instructions leave you with an unconfigured loader, when queuing a task from the interactive shell.

If you had a transport issue, you would have received an error when running the celery server, not when queuing the task.

This is what worked for me:

>>> import djcelery
>>> djcelery.setup_loader()
>>> from myapp.tasks import add
>>> add.delay(2,2)
<AsyncResult: d43a673a-c2a5-4116-b60e-c59afa7dff51>
>>> result = add.delay(4,5)
>>> result.ready()
True
>>> result.result
9
>>> result.get()
9
>>> result.successful()
True