Single SQS Queue vs Multiple SQS Queue while creating a Async Model

Question

I have to develop a component where the Apis are async in nature. In order to develop this async model, I am going to use Aws SQS queues for publishing messages and the client will read from the queue and send the response back into the queue. Now there are 10 APIs (currently) that I have to expose. Currently, I can think of having a single request and a single response queue (which I will poll) for all the APIs and the payload of the APIs can be defined by some Operation. The other way is to use a separate queue for each API. The advantage that I can see for multiple queues is that each API can have different traffic and having multiple queues can help the client of the queues to scale effectively. What can be other pros or cons for both the approaches?

Answer 1

Separate your use-case into 2 distinct problems:

Problem 1: APIs to Workers, one queue or multiple?

If your workers do different types of work, then having a single queue will require them to inspect then discard messages they don't care about. If this is the case, then you should have one queue per message type. This way, any message a worker receives from the queue, it should be able to handle.

If you start ignoring messages, then other workers, who may be idle, may be waiting for a while for messages it cares about.

Problem 2: Using a return queue for the "results". If your clients will be polling for results, then at each poll, your API will need to poll the queue. Again, it will be "searching" for the right response, discarding those it doesn't care about, starving other clients.

Recommendation:

Use multiple queues, one per "worker type". Workers should be able to process any message it receives from the queue.

Then use something other than SQS to store the result. One option is to use S3 to store the result:

1. When your API "creates" the task, create an object in S3 and put a reference to that S3 object on your SQS queue.
2. Your worker will do the work, then put the result where it was told to.
3. When your client polls your API for the result, your API will check S3 and return the status/results.

Instead of S3, other data stores could be used if appropriate: RDS, DynamoDB, etc.