Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to implement a priority queue using SQS(Amazon simple queue service)

I have a situation when a msg fails and I would like to replay that msg with the highest priority using python boto package so he will be taken first. If I'm not wrong SQS queue does not support priority queue, so I would like to implement something simple.

Important note: when a msg fails I no longer have the message object, I only persist the receipt_handle so I can delete the message(if there was more than x retries) / change timeout visibility in order to push him back to queue.

Thanks!

like image 735
OriK Avatar asked Mar 10 '15 17:03

OriK


People also ask

Can you set priority in SQS queue?

As far as I know AWS SQS does not provide a native way or doing a priority queue (single queue priority). If you are open to considering other options, RabbitMQ can do this. Client can specify a priority of 0-255 in the message, and the queue will prioritize a higher priority message gets to the customer first.

What is simple queue service SQS and for what is it used?

Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications.

How does Amazon Simple Queue Service Amazon SQS deliver messages?

Q: Does Amazon SQS guarantee delivery of messages? Standard queues provide at-least-once delivery, which means that each message is delivered at least once. FIFO queues provide exactly-once processing, which means that each message is delivered once and remains available until a consumer processes it and deletes it.


2 Answers

I don't think there is any way to do this with a single SQS queue. You have no control over delivery of messages and, therefore, no way to impose a priority on messages. If you find a way, I would love to hear about it.

I think you could possibly use two queues (or more generally N queues where N is the number of levels of priority) but even this seems impossible if you don't actually have the message object at the time you determine that it has failed. You would need the message object so that the data could be written to the high-priority queue.

I'm not sure this actually qualifies as an answer 8^)

like image 97
garnaat Avatar answered Oct 23 '22 11:10

garnaat


As far as I know AWS SQS does not provide a native way or doing a priority queue (single queue priority). If you are open to considering other options, RabbitMQ can do this. Client can specify a priority of 0-255 in the message, and the queue will prioritize a higher priority message gets to the customer first.

For more information, please look at https://www.rabbitmq.com/priority.html

like image 27
Average Joe Avatar answered Oct 23 '22 10:10

Average Joe