Menu
When would I use a message queue like ironMQ and when would i use a job processing worker like ironWorker?
I have just started researching into these two topics and I am finding it hard to distinguish between the two uses. I understand a worker is more or less a sandbox that will run a program in a different environment outside of the app server to increase the user experience. I also understand that a message queue is much like its database alternative whereby a task is added to a queue and then another server/programming listens for that task and will then process it. However, although I think I understand what they are I am having trouble distinguishing when I would use each one and why.
If I understand correctly I would use a worker for a task such as image processing. But then why can't I use a message queue for that and more importantly why not? Surely I could just have an image URL queued in ironMQ and then have another programming retrieve and process it. In my mind that seems like an extra step so I would avoid that.
A message queue seems fairly pointless to me for common tasks when a worker is available. Surely for non-intensive tasks like posting a comment I could have a worker do that?
I may have misconceived the difference between each tool and if so please set me straight. Otherwise, please help.
516
Queues make your data persistent, and reduce the errors that happen when different parts of your system go offline. By separating different components with message queues, you create more fault tolerance. If one part of the system is ever unreachable, the other can still continue to interact with the queue.
A Message Queue is a mechanism for sharing information, between processes, threads, systems. An AppEngine task Queue is a way for an AppEngine application to say to itself, I need to do this, but I am going to do it later, outside of the context of a client request.
Message queues provide a means for applications to push information to the queue. The message broker simply takes the information from the sender, translates it between different messaging protocols, if needed, and delivers the message to the correct receiver.
In a message-queue model, the publisher pushes messages to a queue where each subscriber can listen to a particular queue. In the event-stream model using Pub/Sub, the publisher pushes messages to a topic that multiple subscribers can listen to.
In modern cloud architecture, applications are decoupled into smaller, independent building blocks that are easier to develop, deploy and maintain. Message queues provide communication and coordination for these distributed applications.
By using a queue between different parts of your software you can decouple the hard dependencies. The format of the message in the queue becomes your data contract and anything that knows how to read that message format can be used to process the transaction.
Another application, called a consumer, connects to the queue and gets the messages to be processed. Messages placed onto the queue are stored until the consumer retrieves them. The queue can provide protection from service outages and failures. Examples of queues: Kafka, Heron, real-time streaming, Amazon SQS, and RabbitMQ.
IBM message queue solutions make life easier for developers by supporting hybrid cloud environments, agile development processes and microservices architectures with an all-in-one messaging backbone. This is especially important as enterprises embark upon application modernization on the journey to cloud. Take the next step:
They are very closely related so I can understand the confusion. They are both queue based systems, one being a message queue, one being a task/job queue. Here's a general rule of thumb:
So no, you don't need a message queue if you're using IronWorker because IronWorker is your message queue + your processing of that queue.
Not to add any confusion, but some people use IronWorker and IronMQ together too, with workers pulling messages off IronMQ. This pattern is good for very short tasks to amortize the setup/teardown of a worker (making database connections or whatever the worker has to do to setup).
155
Message queue is useful when you want to send some data to other process, part of application, or, even, different application. It works like pipe, which transfer data to other side. For example, I have 10 online shops where customers buy things. I want to process all the checkouts on one server. There are some ways to connect shops to processing center:
Cloud services, like IronMQ, do infrastructure support, provide simple libraries for many languages, have great web-based UI, etc.
The same for IronWorker, etc. systems for asynchronous processing. For example, if I have small app, I can do such jobs right on my server. It only requires to use specialized library. In case of medium app, I can setup server and install software for asynchronous tasks processing. If I have huge app with a lot of tasks, it requires to support scalable infrastructure, tasks load balancer, etc.
Companies which provide cloud services for developers are programmed infrastructure controller layer over services like AWS. They provide easy API to their services, support infrastructure and provide a lot of client libraries for different languages.
32
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
