Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

AWS SQS FIFO - How to get more than 10 messages at a time?

Currently we want to pull down an entire FIFO queue, and process the contents, and if any issues, release messages back into the queue.

The problem is, that currently AWS only gives us 10 messages, and won't give us 10 more (which is the way you get bulk messages in SQS, multiple 10 max message requests) until we delete or release the first 10.

We need to get more than 10 though. Is this not possible? We understand we can set the group_id to a random string, and that allows processing more, but then the order isn't guaranteed, which defeats the purpose of FIFO.

like image 438
yellowarchangel Avatar asked Aug 02 '18 20:08

yellowarchangel


People also ask

How many messages can be read from SQS at once?

A single Amazon SQS message queue can contain an unlimited number of messages. However, there is a quota of 120,000 for the number of inflight messages for a standard queue and 20,000 for a FIFO queue.

How we can send more than 256kb message in SQS?

To send messages larger than 256 KB, you can use the Amazon SQS Extended Client Library for Java . This library allows you to send an Amazon SQS message that contains a reference to a message payload in Amazon S3. The maximum payload size is 2 GB. The default visibility timeout for a message is 30 seconds.

Can SQS deliver a message more than once?

Amazon SQS is engineered to provide “at least once” delivery of all messages in its queues. Although most of the time each message will be delivered to your application exactly once, you should design your system so that processing a message more than once does not create any errors or inconsistencies.

How large SQS messages can be what happens when that limit is exceeded?

Each SQS queue is limited to 120,000 inflight messages, or 20,000 if it is a FIFO queue. When sending a message to a queue with too many inflight messages, SQS returns the "OverLimit" error message.


1 Answers

I managed to reproduce your results -- I could retrieve 10 messages, but then running the same command again would not return another set of messages.

The relevant documentation seems to be:

While messages with a particular MessageGroupId are invisible, no more messages belonging to the same MessageGroupId are returned until the visibility timeout expires. You can still receive messages with another MessageGroupId as long as it is also visible.

I suspect (just a theory!) that this is to preserve the ordering of messages... If a client asked for a set of messages and they are still being processed, there is the chance that the messages might be returned to the queue. Therefore, no further messages are provided until the original messages are deleted or pass their visibility timeout.

This is only a behaviour of FIFO queues.

It seems that you will need to receive and delete all messages to be able to access them all. I would suggest:

  • Receive one (or more) message.
  • Process it. If everything worked, delete the message.
  • If there were problems, push the message to a new queue.
  • Once the queue is empty, you would need to read from the new queue and send them back to the original queue (which should preserve ordering).

If you frequently require more capabilities that Amazon SQS provides, you could consider using Amazon MQ – Managed message broker service for ActiveMQ. It has many more capabilities (but is accordingly less 'simple').

like image 160
John Rotenstein Avatar answered Sep 28 '22 10:09

John Rotenstein