Amazon SQS queue limits

Question

My project requires me to communicate with many devices outside the cloud. If successful, this means millions of devices. These devices will not be running Android or iOS, and will be running behind routers & firewalls (I cannot assume they have an external IP).

I am looking to use SQS to send messages to my users outside the cloud. To allow the servers to message individual users, I am designing the system to have one queue per client. This can potentially mean millions (billions?) of queues. While it states that SQS can support unlimited queues, I would like to make sure that I am not abusing the system. If successful, the probability of millions of users is very high.

• I am aware that SQS can be expensive, but I am using it at this stage for ease of administration.
• As far as I can tell SNS requires either an IOS/Android client, or an HTTP server running on the consumer. This is why I ruled out SNS, and I am using SQS.
• I am going to build a distributed cloud front-end over SQS to handle client connections. This front-end will just be a wrapper, that will
  authenticate clients, and relay them to the SQS queues.

Am I abusing the SQS "unlimited queues" policy (will SQS performance drop)? Is there a simpler design for per device messaging?

Answer 1

Let me break the answer by the parts of your question:

About your questions:

Am I abusing the SQS "unlimited queues" policy?

AWS services are designed to prevent abuse and you will pay exactly for what you use, so if you believe this is the right approach, go for it. To remove the uncertainty, i'd advise for a preliminary "proof of concept" implementation.

Is there a simpler design for per device messaging?

Probably yes, re-consider SNS and other messaging systems.

About your statements:

I am aware that SQS can be expensive, but I am using it at this stage for ease of administration.

"Expensive" is a very context-depend classification, considering that a SQS message can cost $0.00000005.

As far as I can tell SNS requires either an IOS/Android client, or an HTTP server running on the consumer. This is why I ruled out SNS, and I am using SQS.

SNS is a push based messaging system (SQS is pull based) that can handle 5 types of subscriptions: smtp, sms, http, mobile push and SQS, so they are not mutually exclusive.

I am going to build a distributed cloud front-end over SQS to handle client connections. This front-end will just be a wrapper, that will authenticate clients, and relay them to the SQS queues.

Managing millions of queues can be a overwhelming task for your "distributed cloud front-end over SQS". Unless your project is exactly about queue management, this is probably undifferentiated heavy lifting.

This is about all i can say without knowing your case, but consider that you can use SNS/SQS together with each other and with other messaging software, such as Apache Camel and others, that may help you build your solution or proof of concept.

Answer 2

I think SQS (or SNS if you can eventually use them) are still your best bet, esp if you need "quick response" or "near real time"; however, for the sake of having "alternatives" just so you can compare...

1. You can consider a giant dynamoDB, with each device/client having it's own "device-id" and perhaps "message-id" as key. This way, your devices can query it's own keys for messages. DynamoDB is meant to handle billions of rows, so this won't stress it much. The querying part, you should be careful though, as you could use up provisioned queries, although at aggregate level, your devices may not all respond/query at the same time, so you may still be ok.

2. You can also consider having a giant S3 bucket, each folder key'ed to the device id and further keyed into message-id folders. This is a poor man's SQS but it's guaranteed to scale, both in message quantity and number of accesses to it.

In both #1 and #2, if your devices are registered with Cognito for credentials, there's a straightforward way to do policies, so the devices can only access their "own" stuff. However, both alternatives #1 and #2 is likely slower than SQS, esp if you use SQS long-poll -- in long poll, you get responses, as soon as SQS detects a message have been dropped off... These alternatives will require you to wait for next cycle-poll.