SQS messages received out of order

Question

I'm using boto3 client in python. I'm pushing messages to sqs but receive them in the wrong order. I can see that their sent time is correct.

The queue is created like that:

boto_session = boto3.Session(region_name=..,aws_access_key_id=..,                                      aws_secret_access_key=...)
sqs_client = boto_session.resource('sqs', endpoint_url=endpoint_url)
sqs_client.create_queue(QueueName=...)

The code to push messages:

boto_session = boto3.Session(region_name=..,aws_access_key_id=..,aws_secret_access_key=...)
sqs_client = boto_session.resource('sqs', endpoint_url=endpoint_url)
queue = sqs_client.get_queue_by_name(QueueName=stream_name)

i = 0
while i < 10:
    print 'b ' + str(i)
    queue.send_message(MessageBody=raw_data.push(json.dumps(dict(id=i)))
    sleep(2)
    i += 1

And code for polling messages:

sqs_resource = boto_session.resource('sqs', endpoint_url=endpoint_url)
queue = sqs_resource.get_queue_by_name(QueueName=queue_name)

while True:
   messages = queue.receive_messages(MaxNumberOfMessages=1,VisibilityTimeout=10,WaitTimeSeconds=5)
   for m in messages:
         print  m.data
         queue.delete_messages(
                Entries=[
                    {
                        'Id': m.message_id,
                        'ReceiptHandle': m.receipt_handle
                    }
                ]
            )

I ran the create queue code, then I pushed messages, then ran a process to consume the messages as shown.

I clearly see the messages are randomly ordered.

Is there a solution for this in sqs? or should I replace the queue?

Answer 1

SQS does not guarantee ordering of messages. It's best to write your code in a way that accepts this or switch to a queue system that does do ordering if you absolutely need it.

Documentation is located here, the relevant passage is:

What if I want the messages to be delivered in order? Amazon SQS makes a best effort to preserve order in messages, but due to the distributed nature of the queue, we cannot guarantee that you will receive messages in the exact order you sent them. You typically place sequencing information or timestamps in your messages so that you can reorder them upon receipt.

Answer 2

When creating the SQS queue please mark it as the FIFO queue, which will guarantee the correct order of the messages.

Please check the Recommendations for FIFO (First-In-First-Out) Queues on the AWS documentation.

Answer 3

If you 100% require a service that retains the order of messages, you could use Amazon Kinesis.

An Amazon Kinesis stream is an ordered sequence of data records. The order of records is retained when data is retrieved, and records can be received by multiple applications. This differs from an Amazon SQS queue, where message order is not guaranteed and, once consumed, messages disappear from the queue.

Consuming messages from Amazon Kinesis is more complex than retrieving messages from an Amazon SQS queue, but it might fit your requirement better.

See: Amazon Kinesis Streams Key Concepts