SQS vs RabbitMQ

Question

I need to create a queue for processing. The queue itself is relatively low-volume. There might be about 1,000 writes to it per hour. The execution of each task might take about a minute each, and are processed almost as soon as the item is added to the queue.

Is there any reason that I might want to implement RabbitMQ instead of something off-the-shelf like Amazon SQS? What are some reasons why an application would need its own queueing system instead of something like SQS?

Answer 1

For a start, Amazon SQS is a pseudo-queue which means the delivery of every message(if it reaches the queue) is guaranteed but not in a FIFO fashion which usually happens in a queue.

If the order of messages is important to you and you want the queue to work in a FIFO fashion, the Amazon SQS documentation states to handle this in your application logic as the messages from the Amazon SQS will reach you out of sequence.

Compared to this, as far as I know, you can implement worker queues in RabbitMQ. If that rids you of implementing queue message sequencing at application level then this would be a more preferable option.

Here are a few factors to help you decide which one to go for:

1. Queue message sequence as mentioned above.

2. You can setup your own server with RabbitMQ but not in the case of Amazon SQS so the cost gets involved here.

3. Setting up your own server will require good knowledge of the subject so that you do not leave any corner untouched. This is not the case with Amazon SQS as it is pretty quick to get started with.

4. Your own RabbitMQ server means maintenance cost down the line which is not the case with Amazon SQS.

Updates:

1. Amazon SQS now supports FIFO queues.

Answer 2

SQS would be my preference over RabbitMQ, here is why.

1. SQS is a managed service. So you don't have to worry about operational aspects of running a messaging system including administration, security, monitoring etc. Amazon will do this for you and will provide support if something were to go wrong.
2. SQS is Elastic and can scale to very large rate/volumes (unlimited according to AWS ;))
3. Availability of SQS has a lot of 9's in it and is backed by Amazon, which is one less thing to worry about in your application.

However RabbitMQ might provide faster response times for puts and gets, typically in 10s of thousands of TPS from my testing. For SQS to provide that kind of throughput, you will have to scale up horizontally with multiple instances. So if you are looking for under 5ms puts , RabbitMQ might be an option to consider because i have seen close to 20ms-30ms put time from my SQS testing at 1000s of TPS, which is slightly higher than RabbitMQ.

We just moved our messaging infrastructure from ActiveMQ to SQS and can't be any more happier. We have found it to be cheaper than maintaining our own ActiveMQ cluster in the cloud.

Hope this helps! Let us know how it goes..