Subscribing to AWS SQS Messages

Question

I have large number of messages in AWS SQS Queue. These messages will be pushed to it constantly by other source. There are no proper dynamic on how often those messages will be pushed to queue. Currently, I keep polling SQS every second and checking if there are any messages available in there. Is there any better way of handling this, like receiving notification from SQS or SNS that some messages are available so that I only request SQS when I needed instead of constant polling?

Answer 1

You can also use the worker functionality in AWS Elastic Beanstalk. It allows you to build a worker to process each message, and when you use Elastic Beanstalk to deploy it to an EC2 instance, you can define it as subscribed to a specific queue. Then each message will be POST to the worker, without your need to call receive-message on it from the queue.

It makes your system wiring much easier, as you can also have auto scaling rules that will allow you to spawn multiple workers to handle more messages in time of peak load, and scale down back to a single worker, when the load is low. It will also delete the message automatically, if you respond with OK from your worker.

See more information about it here: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html

Answer 2

The way to do what you want is to use long polling - rather than constantly poll every second, you open a request that stays open until it either times out or a message comes into the queue. Take a look at the documentation for ReceiveMessageRequest

ReceiveMessageRequest req = new ReceiveMessageRequest()
    .withWaitTimeSeconds(Integer.valueOf(20));    // set long poll timeout to 20 sec
// set other properties on the request as well

ReceiveMessageResult result = amazonSQS.receiveMessage(req);

A common usage pattern for this is to have a background thread running the long poll and pushing the results into an internal queue (such as LinkedBlockingQueue or an ExecutorService) for a worker thread to read from.

PS. Don't forget to call deleteMessage once you're done processing the result so you don't end up receiving it again.

Answer 3

You could also have a look at Shoryuken and the property delay:

delay: 25 # The delay in seconds to pause a queue when it's empty

But being honest we use delay: 0 here, the cost of SQS is inexpensive:

• First 1 million Amazon SQS Requests per month are free
• $0.50 per 1 million Amazon SQS Requests per month thereafter ($0.00000050 per SQS Request)
• A single request can have from 1 to 10 messages, up to a maximum total payload of 256KB.
• Each 64KB ‘chunk’ of payload is billed as 1 request. For example, a single API call with a 256KB payload will be billed as four requests.

You will probably spend less than 10 dollars monthly polling messages every second 24x7 in a single host.

One of the advantages of Shoryuken is that it fetches in batch, so it saves some money compared with a fetch per message solutions.