Is it possible to listen to AWS SNS notifications without using public callbacks?

Question

I'm just getting started with SNS and as far as I see you cannot simply subscribe to the publisher and listen to events. You have to i). create a publically accessible callback (url/email/sms), ii). register it with the service and iii). build a consumer specific to the type of callback.

I want to use an API similar to this:

const client = new SnsClient({
  region: 'eu-west-1',
  topicArn: 'XXX'
})
client.on('connection', (connection) => {
  connection.on('notification', (notification) => {
    // do some work with notification
  })
})

Am I missing something? I want to have X number of web servers listening to a single event (fanout).

For instance my worker might be an EC2 instance on a private subnet, so the available subscriptions: "http(s)", "email", "sms" wouldn't work**. "SQS" could work but you have to set a queue up for each instance and it uses long polling rather than push. "application" and "lambda" are not applicable.

Is AWS SNS suitable for this use case? If not is there an alternative AWS service?

** You might be able to get https to work but only with overly complex roles/dns.

Edit: I think what I want to do is similar to Google Cloud PubSub Subscription or RabbitMQ but using a native (AWS) rather than 3rd party service.

Answer 1

"SQS" could work but you have to set a queue up for each instance and it uses long polling rather than push.

SQS is exactly the way to implement what you have described, in AWS. SNS does not support a connection.on('notification',... style of operation, but this is exactly how SQS long polling works, in practice.

Don't be confused by the phrase "long polling." Yes, technically that's what it is, but it's very much a legitimate push operation that occurs over the long poll -- if you're, say, 5 seconds into a 20 second long poll against an empty queue, the next message that arrives will immediately cause your long poll to return that one message. Now. Not in another 15 seconds. You don't sit around and wait for more, even if you asked for more. Long polling is essentially a push-wrapper in SQS.

Note that you don't necessarily have to define the fan-out queues up front. Each listener can create a queue for itself, subscribe the queue to the topic, and start listening.

Or... there's the message broker in AWS IoT. Some might say this is a little unorthodox of an application, but it appears to support multiple subscribers to one topic.

The message broker maintains a list of all client sessions and the subscriptions for each session. When a message is published on a topic, the broker checks for sessions with subscriptions that map to the topic. The broker then forwards the publish message to all sessions that have a currently connected client.

http://docs.aws.amazon.com/iot/latest/developerguide/iot-message-broker.html

It supports MQTT over TCP, and MQTT over web sockets, with TLS in both cases.