Using Celery with SQLAlchemy and Pyramid

Question

I'm creating the web app using Pyramid-1.2.1 with SQLAlchemy as database backend.
Now I need to do some periodic tasks along with this app and I want to use Celery as a task processor and SQLAlchemy as the message queue and the result backend.

1) I've installed Celery-2.1.4 but I can not figure out how to make it work. Here is my celeryconfig.py file:

# List of modules to import when celery starts.
CELERY_IMPORTS = ("my_app.tasks", )

## Result store settings.
CELERY_RESULT_BACKEND = "database"
CELERY_RESULT_DBURI = "sqlite:///MyDataBase.db"
CELERY_RESULT_SERIALIZER = "json"

## Broker settings.
BROKER_TRANSPORT = "sqlakombu.transport.Transport"
BROKER_HOST = "sqlite:///MyDataBase.db"

The my_app.tasks contain a simple addition task from the celery examples.
Now when I run

$ celeryd -l info

I see the following:

[2011-11-11 20:22:50,750: WARNING/MainProcess] [email protected] v2.1.4 is starting.
[2011-11-11 20:22:50,765: WARNING/MainProcess]  
Configuration ->
    . broker -> sqlakombu.transport.Transport://guest@sqlite:///MyDataBase.db/
    . queues ->
        . celery -> exchange:celery (direct) binding:celery
    . concurrency -> 4
    . loader -> celery.loaders.default.Loader
    . logfile -> [stderr]@INFO
    . events -> OFF
    . beat -> OFF
    . tasks ->
        . chatrooms.task.add
[2011-11-11 20:22:50,787: INFO/PoolWorker-1] child process calling self.run()
[2011-11-11 20:22:50,789: INFO/PoolWorker-2] child process calling self.run()
[2011-11-11 20:22:50,791: INFO/PoolWorker-3] child process calling self.run()
[2011-11-11 20:22:50,796: INFO/PoolWorker-4] child process calling self.run()
[2011-11-11 20:22:50,802: WARNING/MainProcess] [email protected] has started.
[2011-11-11 20:22:50,804: WARNING/MainProcess] Traceback (most recent call last):
[2011-11-11 20:22:50,805: WARNING/MainProcess] File "/Users/shashkin/python_v_env/bin/celeryd", line 8, in <module>
[2011-11-11 20:22:50,805: WARNING/MainProcess] load_entry_point('celery==2.1.4', 'console_scripts', 'celeryd')()
[2011-11-11 20:22:50,805: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/celery-2.1.4-py2.7.egg/celery/bin/celeryd.py", line 166, in main
[2011-11-11 20:22:50,805: WARNING/MainProcess] worker.execute_from_commandline()
[2011-11-11 20:22:50,806: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/celery-2.1.4-py2.7.egg/celery/bin/base.py", line 40, in execute_from_commandline
[2011-11-11 20:22:50,806: WARNING/MainProcess] return self.run(*args, **vars(options))
[2011-11-11 20:22:50,806: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/celery-2.1.4-py2.7.egg/celery/bin/celeryd.py", line 85, in run
[2011-11-11 20:22:50,806: WARNING/MainProcess] return Worker(**kwargs).run()
[2011-11-11 20:22:50,806: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/celery-2.1.4-py2.7.egg/celery/apps/worker.py", line 121, in run
[2011-11-11 20:22:50,807: WARNING/MainProcess] self.run_worker()
[2011-11-11 20:22:50,807: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/celery-2.1.4-py2.7.egg/celery/apps/worker.py", line 219, in run_worker
[2011-11-11 20:22:50,807: WARNING/MainProcess] worker.start()
[2011-11-11 20:22:50,807: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/celery-2.1.4-py2.7.egg/celery/worker/__init__.py", line 217, in start
[2011-11-11 20:22:50,808: WARNING/MainProcess] component.start()
[2011-11-11 20:22:50,808: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/celery-2.1.4-py2.7.egg/celery/worker/listener.py", line 238, in start
[2011-11-11 20:22:50,808: WARNING/MainProcess] self.reset_connection()
[2011-11-11 20:22:50,808: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/celery-2.1.4-py2.7.egg/celery/worker/listener.py", line 416, in reset_connection
[2011-11-11 20:22:50,808: WARNING/MainProcess] self.connection = self._open_connection()
[2011-11-11 20:22:50,808: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/celery-2.1.4-py2.7.egg/celery/worker/listener.py", line 480, in _open_connection
[2011-11-11 20:22:50,809: WARNING/MainProcess] max_retries=conf.BROKER_CONNECTION_MAX_RETRIES)
[2011-11-11 20:22:50,809: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/celery-2.1.4-py2.7.egg/celery/utils/__init__.py", line 276, in retry_over_time
[2011-11-11 20:22:50,809: WARNING/MainProcess] retval = fun(*args, **kwargs)
[2011-11-11 20:22:50,809: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/celery-2.1.4-py2.7.egg/celery/worker/listener.py", line 472, in _establish_connection
[2011-11-11 20:22:50,809: WARNING/MainProcess] conn.connect()                              # evaluate connection
[2011-11-11 20:22:50,809: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/carrot-0.10.7-py2.7.egg/carrot/connection.py", line 170, in connect
[2011-11-11 20:22:50,810: WARNING/MainProcess] return self.connection
[2011-11-11 20:22:50,810: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/carrot-0.10.7-py2.7.egg/carrot/connection.py", line 135, in connection
[2011-11-11 20:22:50,810: WARNING/MainProcess] self._connection = self._establish_connection()
[2011-11-11 20:22:50,810: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/carrot-0.10.7-py2.7.egg/carrot/connection.py", line 148, in _establish_connection
[2011-11-11 20:22:50,810: WARNING/MainProcess] return self.create_backend().establish_connection()
[2011-11-11 20:22:50,810: WARNING/MainProcess] File "/Users/shashkin/python_v_env/lib/python2.7/site-packages/carrot-0.10.7-py2.7.egg/carrot/connection.py", line 161, in create_backend
[2011-11-11 20:22:50,810: WARNING/MainProcess] return backend_cls(connection=self)
[2011-11-11 20:22:50,811: WARNING/MainProcess] TypeError
[2011-11-11 20:22:50,811: WARNING/MainProcess] :
[2011-11-11 20:22:50,811: WARNING/MainProcess] __init__() takes exactly 2 arguments (1 given)
[2011-11-11 20:22:50,811: INFO/MainProcess] process shutting down

What am I missing?

2) Later I want to start Celery inside my app. What should I do after celery-pylons package installation? Should I put celery's settings in app's .ini file? Could someone provide me any example, because those I found in the net did not help me much.

Thanks.

Answer 1

This looks like an issue with version compatibility between celery packages and its dependencies. With the latest version of celery, 2.4.5 your example just works. With 2.1.4 I get the same error.

If for any reason you need 2.1.4 version of celery only, you have to find out which dependencies you need to downgrade in order to make it work. By looking at the traceback, most likely, it'll be the carrot dependency but you may get into a dependency hell if other dependencies need certain version of carrot.

I strongly suggest to use virtualenv with pip to be able to experiment with the versions of the packages easily. Trying to develop something using Python packages from Linux repositories can be very painful.

buildout versions or pip requirements address this kind of problems but AFAIK there is no set of pinned dependency versions for celery.