NodeJS and AWS SQS

Question

Folks, I would like to set up a message queue between our Java API and NodeJS API. After reading several examples of using aws-sdk, I am not sure how to make the service watch the queue.

For instance, this article Using SQS with Node: Receiving Messages Example Code tells me to use the sqs.receiveMessage() to receive and sqs.deleteMessage() to delete a message.

What I am not clear about, is how to wrap this into a service that runs continuously, which constantly takes the messages off the sqs queue, passes them to the model, stores them in mongo, etc.

Hope my question is not entirely vague. My experience with Node lies primarily with Express.js.

Is the answer as simple as using something like sqs-poller? How would I implement the same into an already running NodeJS Express app? Quite possibly I should look into SNS to not have any delay in message transfers.

Thanks!

Answer 1

For a start, Amazon SQS is a pseudo queue that guarantees availability of messages but not their sequence in FIFO fashion. You have to implement sequencing logic into your app if you want it to work that way.

Coming back to your question, SQS has to be polled within your app to check if there are new messages available. I implemented this in an app using setInterval(). I would poll the queue for items and if no items were found, I would delay the next call and in case some items were found, the next call would be immediate bypassing the setInterval(). This is obviously a very raw implementation and you can look into alternatives. How about a child process on your server that pings your NodeJS app when a new item is found in SQS ? I think you can implement the child process as a watcher in BASH without using NodeJS. You can also look into npm modules if there is already one for this.

In short, there are many ways you can poll but polling has to be done one way or the other if you are working with Amazon SQS.

I am not sure about this but if you want to be notified of items, you might want to look into Amazon SNS.

Answer 2

When writing applications to consume messages from SQS I use sqs-consumer:

const Consumer = require('sqs-consumer');

const app = Consumer.create({
  queueUrl: 'https://sqs.eu-west-1.amazonaws.com/account-id/queue-name',
  handleMessage: (message, done) => {
    console.log('Processing message: ', message);
    done();
  }
});

app.on('error', (err) => {
  console.log(err.message);
});

app.start();

See the docs for more information (well documented): https://github.com/bbc/sqs-consumer