Running multiple Django Celery websites on same server

Question

I'm running multiple Django/apache/wsgi websites on the same server using apache2 virtual servers. And I would like to use celery, but if I start celeryd for multiple websites, all the websites will use the configuration (logs, DB, etc) of the last celeryd instance I started.

Is there a way to use multiple Celeryd (one for each website) or one Celeryd for all of them? Seems like it should be doable, but I can't find out how.

Answer 1

This problem was a big headache, i didn't noticed @Crazyshezy 's comment when i first came here. I just accomplished this by changing Broker URL in settings.py for each web app.

app1.settings.py

BROKER_URL = 'redis://localhost:6379/0'

app2.settings.py

BROKER_URL = 'redis://localhost:6379/1'

Answer 2

Yes there is a way. We use supervisor to start celery daemons for every project we need it.

The supervisor config file looks something like this:

[program:PROJECTNAME]
command=python manage.py celeryd --loglevel=INFO --beat
environment=PATH=/home/www-data/projects/PROJECTNAME/env/bin:/usr/bin:/bin
directory=/home/www-data/projects/PROJECTNAME/
user=www-data
numprocs=1
umask=022
stdout_logfile=/home/www-data/logs/%(program_name)s.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=10
stderr_logfile=/home/www-data/logs/%(program_name)s.error.log
stderr_logfile_maxbytes=50MB
stderr_logfile_backups=10
autorestart=true
autostart=True
startsecs=10
stopwaitsecs = 60
priority=998

There is also an other advantage if you use this setup: The celery daemons run entirely in userspace.

Remember to use different broker backends for your projects. It won't work if you use the same rabbitmq virtualhost or if you use the same redis database for every project.