Read Celery configuration from Python properties file

Question

I have an application that needs to initialize Celery and other things (e.g. database). I would like to have a .ini file that would contain the applications configuration. This should be passed to the application at runtime.

development.init:

[celery]
broker=amqp://localhost/
backend=amqp://localhost/
task.result.expires=3600

[database]
# database config
# ...

celeryconfig.py:

from celery import Celery
import ConfigParser

config = ConfigParser.RawConfigParser()
config.read(...) # Pass this from the command line somehow

celery = Celery('myproject.celery',
                broker=config.get('celery', 'broker'),
                backend=config.get('celery', 'backend'),
                include=['myproject.tasks'])

# Optional configuration, see the application user guide.
celery.conf.update(
    CELERY_TASK_RESULT_EXPIRES=config.getint('celery', 'task.result.expires')
)

# Initialize database, etc.

if __name__ == '__main__':
    celery.start()

To start Celery, I call:

celery worker --app=myproject.celeryconfig -l info

Is there anyway to pass in the config file without doing something ugly like setting a environment variable?

Answer 1

Alright, I took Jordan's advice and used a env variable. This is what I get in celeryconfig.py:

celery import Celery
import os
import sys
import ConfigParser

CELERY_CONFIG = 'CELERY_CONFIG'

if not CELERY_CONFIG in os.environ:
    sys.stderr.write('Missing env variable "%s"\n\n' % CELERY_CONFIG)
    sys.exit(2)

configfile = os.environ['CELERY_CONFIG']

if not os.path.isfile(configfile):
    sys.stderr.write('Can\'t read file: "%s"\n\n' % configfile)
    sys.exit(2)

config = ConfigParser.RawConfigParser()
config.read(configfile)

celery = Celery('myproject.celery',
                broker=config.get('celery', 'broker'),
                backend=config.get('celery', 'backend'),
                include=['myproject.tasks'])

# Optional configuration, see the application user guide.
celery.conf.update(
    CELERY_TASK_RESULT_EXPIRES=config.getint('celery', 'task.result.expires'),
)

if __name__ == '__main__':
    celery.start()

To start Celery:

$ export CELERY_CONFIG=development.ini
$ celery worker --app=myproject.celeryconfig -l info

Answer 2

How is setting an environment variable ugly? You either set an environment variable with the current version of your application, or you can derive it based on hostname, or you can make your build/deployment process overwrite the file, and on development you let development.ini copy over to settings.ini in a general location, and on production you let production.ini copy over to settings.ini.

Any of these options are quite common. Using a configuration management tool such as Chef or Puppet to put the file in place is a good option.