Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

AWS - SQS Batch Size and Lambda approach

If I understand correctly, batch size setting on Lambda decides how many messages to take in one sweep from the SQS. Therefore this JSON (taken from the test Lambda SQS);

{
  "Records": [
    {
      "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
      "receiptHandle": "MessageReceiptHandle",
      "body": "FAIL",
      "attributes": {
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1523232000000",
        "SenderId": "123456789012",
        "ApproximateFirstReceiveTimestamp": "1523232000001"
      },
      "messageAttributes": {
      },
      "md5OfBody": "7b270e59b47ff90a553787216d55d91d",
      "eventSource": "aws:sqs",
      "eventSourceARN": "arn:aws:sqs:eu-west-1:123456789012:MyQueue",
      "awsRegion": "eu-west-1"
    }
  ]
}

There is Records array. And if I set batch size to 5, then if there are 5 messages in SQS, they will be included in array. If there are 10 messages, then Lambda will be invoked twice, with 5 messages in each Record.

I am now confused a bit, as to what approach to take. My Lambda is fairly simple. It will be Axios POST request to external service. If it errors out, I will throw an error. I could even use axios-retry, and make retries fairly easy.

Should I use batch in my case? Naively lookin, all I need is 1 to 1. In other words, message arrives. Lambda takes it. If it errors, it will be retried automatically a bit later.

Contrary, I would have to iterate via all messages and attempt an Axios request. What if the third of five messages fails, in that case I throw an error and Lambda is stopped. What happens to messages four and five? Are they resent to SQS and then again picked up for another execution?

like image 894
Wexoni Avatar asked Feb 07 '19 13:02

Wexoni


People also ask

What is a batch size in SQS Lambda?

Batch size – The number of records to send to the function in each batch. For a standard queue, this can be up to 10,000 records. For a FIFO queue, the maximum is 10.

How many SQS messages can Lambda process?

By default, Lambda batches up to 10 messages in a queue to process them during a single Lambda execution. You can increase this number up to 10,000 messages, or up to 6 MB of messages in a single batch for standard SQS queues.

Can Lambda listen to multiple SQS?

Yes there's no reason you can't configure it that way.


1 Answers

all I need is 1 to 1. In other words, message arrives. Lambda takes it. If it errors, it will be retried automatically a bit later.

I don't think you need batch processing in above case.

What if the third of five messages fails, in that case I throw an error and Lambda is stopped. What happens to messages four and five? Are they resent to SQS and then again picked up for another execution?

Message(s) won't be deleted from the queue if your Lambda errors, based on the SQS visibility timeout and redrive policy configuration. SQS will trigger Lambda again according to the configuration. If you have configured DLQ then after maxReceiveCount is reached, failed messages will be added to the DLQ and removed from the main queue.

like image 86
A.Khan Avatar answered Sep 24 '22 01:09

A.Khan