Amazon SQS, Is it possible to fetch messages from SQS on basis of Sender ID

Question

We have multiple Amazon Users, which are sending messages into same SQS Queue, So is it possible to fetch message from queue on Basis of Sender ID? means to return specific messages list on basis of Sender ID.

Answer 1

No, you can't filter your message get request on senderid, you would need to read all the messages and 'throw out' (return) the ones that don't match, but that would not be the best way to go about it - you'll be paying for all those read requests and throwing out lots of messages (i.e. returning them to the queue).

While all of the 'wrong' consumers are doing this, the consumer who actually wants/needs messages assigned to them are going to be blocked from actually getting their own messages - possibly for a very long time.

Not sure of your intent, but it sounds like you would be better off using multiple queues, one per senderid, if that is how you want to consume the messages - there is no extra cost to created extra queues.

Answer 2

I am assuming there would be a need to process the queue items [entities] after the filtering operation is completed. While multiple queues can certainly help, the more robust way is to perform the pre-processing done, then an there [like filtering by sender id] in my opinion an elegant way.

With the release of AWS Lambda, you can certainly offload the filtering or pre-processing and then push / place it at appropriate places.

If you have your users who push the items to Queue, then try the following below approach.

You can try with 2 Lambda Functions [Feeder & Worker].

Feeder would be scheduled lambda function whose job is to take items from SQS (if any) and push it as an SNS topic (and continue doing it forever)

Worker would be linked to listen the SNS topic which would do the data filtering [depending on the sender ID]. The passed items can be move to a brand new queue and the rest can be scrapped.

The better way would be to have your users push the entities directly to SNS topic and lambda function can perform the filtering directly.