Why does synchronous lambda invoke multiple times?

Question

I have a lambda function my-func which should only be run once at a time. Because of that, I set the Reserved Concurrency to 1. I am trying to invoke it with the command:

aws lambda invoke --function-name my-func --invocation-type RequestResponse --cli-binary-format raw-in-base64-out --payload '{\"recreate\":true}' response.json

However, it results in this error after a few seconds:

An error occurred (TooManyRequestsException) when calling the Invoke operation (reached max retries: 2): Rate Exceeded.

It appears that it tries to invoke the function multiple times even though the original invocation never ran into an error. If I increase the Reserved Concurrency to a value like 5, then the single lambda invoke command results in multiple invocations even though the first invocation continues to execute without any problem.

Another thing that is throwing me off is that it works correctly from the AWS console GUI. I created a test event in the AWS lambda function on the console interface. It invokes my-func with the same payload I am using in the aws-cli command:

{
  "recreate": true
}

Invoking the function using this test event works flawlessly. It seems to just run the function once and doesn't cause a TooManyRequestsException. Does this mean something is wrong with my aws-cli command?

Answer 1

I've figured out the problem. Even though my function's timeout was set to 600 (5 minutes), the aws cli has it's own socket timeout which defaults to 60 seconds. Each time it reached this timeout, it must have triggered a retry. I fixed it by adding --cli-read-timeout 600 to my command like so:

aws --cli-read-timeout 600 lambda invoke --function-name my-func --invocation-type RequestResponse --cli-binary-format raw-in-base64-out --payload '{\"recreate\":true}' response.json