Menu
From my understanding, Celery is a distributed task queue, which means the only thing that it should do is dispatching tasks/jobs to others servers and get the result back. RabbitMQ is a message queue, and nothing more. However, a worker could just listen to the MQ and execute the task when a message is received. This achieves exactly what Celery offers, so why need Celery at all?
230
From my understanding, Celery is a distributed task queue, which means the only thing that it should do is dispatching tasks/jobs to others servers and get the result back. RabbitMQ is a message queue, and nothing more. However, a worker could just listen to the MQ and execute the task when a message is received.
It's incredibly lightweight, supports multiple brokers (RabbitMQ, Redis, and Amazon SQS), and also integrates with many web frameworks, e.g. Django, etc. Celery's asynchronous task queue allows the execution of tasks and its concurrency makes it useful in several production systems.
Celery allows Python applications to quickly implement task queues for many workers. It takes care of the hard part of receiving tasks and assigning them appropriately to workers. You use Celery to accomplish a few main goals: Define independent tasks that your workers can do as a Python function.
Message broker such as RabbitMQ provide communication between nodes. Running your Celery clients, workers, and related broker in the cloud gives your team the power to easily manage and scale backend processes, jobs, and basic administrative tasks.
You are right, you don't need Celery at all. When you are designing a distributed system there are a lot of options and there is no right way to do things that fits all situations.
Many people find that it is more flexible to have pools of message consumers waiting for a message to appear on their queue, doing some work, and sending a message when the work is finished.
Celery is a framework that wraps up a whole lot of things in a package but if you don't really need the whole package, then it is better to set up RabbitMQ and implement just what you need without all the complexity. In addition, RabbitMQ can be used in many more scenarios besides the task queue scenario that Celery implements.
But if you do choose Celery, then think twice about RabbitMQ. Celery's message queueing model is simplistic and it is really a better fit for something like Redis than for RabbitMQ. Rabbit has a rich set of options that Celery basically ignores.
167
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
