Priority queue with AWS SQS and Lambda as consumer

Question

I have a messaging use case question.

As of now, we have a queue in AWS SQS, say, origQueue and out-of-the-box-lambda-based-message-consumption on that queue.

Now to cater to one particular feature of priority (High,Medium,Low) based message consumption (on the basis of a 'priority' number set within the message), I am thinking to have a set of 3 queues, wherein each queue is pertaining to a different priority level. On the highest priority queue, out-of-the-box-lambda-based-message-consumption would continue to happen. A batch process would keep running in an interval of 5 mins. each to promote some messages from the mid and low priority queues. The logic of this batch process has not been thought of currently, but it could be anything, say pick up 10 Medium priority messages and 5 low priority messages , both aged more than 1 hour and promote them to the high priority queue, so that they can be consumed by the above mentioned out-of-the-box-lambda-based-message-consumption.

So before going that way, I just wanted to gather other potential ideas. Is there any out-of-the-box AWS feature or any pattern to solve this priority based message consumption problem?

P.S.

Another (not chosen) approach I came up with was to 'insert' the items in the queue considering the priority which would keep the queue always ordered by priority. But this 'run-time-dynamic-insertion' does not seem feasible as the stream of incoming messages is always on.

Answer 1

There is now the concept of Message Groups for FIFO (First in, First Out) SQS queues. This allows you to send a message to the SQS queue with a particular ID, in which the queue will automatically group it for you in the order it receives new items.

Interacting code can query the overall SQS queue with a Message Group parameter and will receive back any messages in that queue. Using something like ReceiveMessage (Node.js link provided), one can provide the parameter "MessageGroupId" which will then fetch one or more messages (in the order received) with a matching "MessageGroupId".

A basic high level concept would be to have the code always checking the high priority message group until it determines it is empty, and then moving to the medium (and then the low) priority queues to process work there until a new entry in a higher priority queue is found.

Answer 2

The AWS recommended way is as you first suggest, to have 3 separate SQS queues, ie one for each priority level. See the suggestion here at the bottom of the page. Whatever consumes each queue will have to be coordinated separately, as none of the queues 'knows' what their priority is inherently.