Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Using Multiple Installations of Celery with a Redis Backend

Is it possible to use the same redis database for multiple projects using celery? Like using the same database for multiple projects as a cache using a key prefix. Or do i have to use a seperate database for every installation?

like image 588
frog32 Avatar asked Aug 21 '12 09:08

frog32


People also ask

How many tasks can Celery Redis handle?

Save this answer. Show activity on this post. celery beats only trigger those 1000 tasks (by the crontab schedule), not run them. If you want to run 1000 tasks in parallel, you should have enough celery workers available to run those tasks.

How Redis works with Celery?

Redis is the datastore and message broker between Celery and Django. In other words, Django and Celery use Redis to communicate with each other (instead of a SQL database). Redis can also be used as a cache as well. An alternative for Django & Celery is RabbitMQ (not covered here).

What is the difference between Celery and Redis?

Celery belongs to "Message Queue" category of the tech stack, while Redis can be primarily classified under "In-Memory Databases". "Task queue" is the primary reason why developers consider Celery over the competitors, whereas "Performance" was stated as the key factor in picking Redis.


1 Answers

To summarise from this helpful blog post: http://kfalck.net/2013/02/21/run-multiple-celeries-on-a-single-redis

  • Specify a different database number for each project, e.g. redis://localhost/0 and redis://localhost/1
  • Define and use different queue names for the different projects. On the task side, define CELERY_DEFAULT_QUEUE, and when starting up your worker, use the -Q parameter to specify that queue. Read more about routing here: http://docs.celeryproject.org/en/latest/userguide/routing.html
like image 73
Charl Botha Avatar answered Oct 13 '22 18:10

Charl Botha