I wonder what is the difference between them. Are they describing the same thing?
Is Google App Engine Service Task Queue is an implementation of Message Queue?
RabbitMQ work queues are used to distribute time-consuming tasks among multiple workers. The main idea behind Work Queues (also called task queues) is to avoid doing a resource-intensive task immediately and having to wait for it to complete.
Message queues provide a means for applications to push information to the queue. The message broker simply takes the information from the sender, translates it between different messaging protocols, if needed, and delivers the message to the correct receiver.
Task queues let applications perform work, called tasks, asynchronously outside of a user request. If an app needs to execute work in the background, it adds tasks to task queues. The tasks are executed later, by worker services. The Task Queue service is designed for asynchronous work.
Celery is an asynchronous distributed task queue. RabbitMQ is a message broker which implements the Advanced Message Queuing Protocol (AMQP).
I asked a similar question on some Developer Community Groups on Facebook. It was not about GoogleAppEngine specifically - i asked in more of a general sense to determine use case between RabbitMQ and Celery. Here are the responses I got which I think is relevant to the topic and fairly clarifies the difference between a message queue and a task queue.
I asked:
Will it be appropriate to say that "Celery is a QueueWrapper/QueueFramework which takes away the complexity of having to manage the internal queueManagement/queueAdministration activities etc"?
I understand the book language which says "Celery is a task queue" and "RabbitMQ is a message broker". However, it seems a little confusing as a first-time celery user because we have always known RabbitMQ to be the 'queue'.
Please help in explaining how/what celery does in constrast with rabbitMQ
A response I got from Abu Ashraf Masnun
Task Queue and Message Queue. RabbitMQ is a "MQ". It receives messages and delivers messages.
Celery is a Task Queue. It receives tasks with their related data, runs them and delivers the results.
Let's forget Celery for a moment. Let's talk about RabbitMQ. What would we usually do? Our Django/Flask app would send a message to a queue. We will have some workers running which will be waiting for new messages in certain queues. When a new message arrives, it starts working and processes the tasks.
Celery manages this entire process beautifully. We no longer need to learn or worry about the details of AMQP or RabbitMQ. We can use Redis or even a database (MySQL for example) as a message broker. Celery allows us to define "Tasks" with our worker codes. When we need to do something in the background (or even foreground), we can just call this task (for instant execution) or schedule this task for delayed processing. Celery would handle the message passing and running the tasks. It would launch workers which would know how to run your defined tasks and store the results. So you can later query the task result or even task progress when needed.
You can use Celery as an alternative for cron job too (though I don't really like it)!
Another response I got from Juan Francisco Calderon Zumba
My understanding is that celery is just a very high level of abstraction to implement the producer / consumer of events. It takes out several painful things you need to do to work for example with rabbitmq. Celery itself is not the queue. The events queues are stored in the system of your choice, celery helps you to work with such events without having to write the producer / consumer from scratch.
Eventually, here is what I took home as my final learning:
Celery is a queue Wrapper/Framework which takes away the complexity of having to manage the underlying AMQP mechanisms/architecture that come with operating RabbitMQ directly
GAE's Task Queues are a means for allowing an application to do background processing, and they are not going to serve the same purpose as a Message Queue. They are very different things that serve different functions.
A Message Queue is a mechanism for sharing information, between processes, threads, systems.
An AppEngine task Queue is a way for an AppEngine application to say to itself, I need to do this, but I am going to do it later, outside of the context of a client request.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With