AWS Lambda + API-gateway return 502 error when the lambda is throttled

Question

Synchronous invocation: If the function is invoked synchronously and is throttled, Lambda returns a 429 error and the invoking service is responsible for retries. The ThrottledReason error code explains whether you ran into a function level throttle (if specified) or an account level throttle (see note below). Each service may have its own retry policy. For example, CloudWatch Logs retries the failed batch up to five times with delays between retries. For a list of event sources and their invocation type, see Supported Event Sources.

Reference

I not sure my understanding about the above sentence is right, If I am wrong please fix me.

1. When a lambda is throttled, it returns 429 Error to API-gateway.
2. The invoking service, in here API-gateway, retries the request.

However, It doesn't work as expected. The below is the API-gateway log from cloudWatch when a lambda is throttled.

API-Gateway-Execution-Logs_3f1frvtwe4/sam-sm-test 2a38a4a9316c49e5a833517c45d31070 (bededbf0-73ae-11e8-87a2-f51933ef104f) Endpoint response body before transformations: {"Reason":"ReservedFunctionConcurrentInvocationLimitExceeded","Type":"User","message":"Rate Exceeded."}
API-Gateway-Execution-Logs_3f1frvtwe4/sam-sm-test 2a38a4a9316c49e5a833517c45d31070 (bededbf0-73ae-11e8-87a2-f51933ef104f) Endpoint response headers: {Connection=keep-alive, x-amzn-RequestId=bedfc624-73ae-11e8-bd28-6345cb3606c4, x-amzn-ErrorType=TooManyRequestsException, Content-Length=104, Date=Tue, 19 Jun 2018 10:51:39 GMT, Content-Type=application/json}
API-Gateway-Execution-Logs_3f1frvtwe4/sam-sm-test 2a38a4a9316c49e5a833517c45d31070 (bededbf0-73ae-11e8-87a2-f51933ef104f) Execution failed due to configuration error: Malformed Lambda proxy response

In practical, Lambda returns {"Reason":"ReservedFunctionConcurrentInvocationLimitExceeded","Type":"User","message":"Rate Exceeded."} which is wrong format for API-gateway(proxy integration) then, as the result, API-gateway returns 502 Error to client calling the API.

I want the failed request to be retried. how could I handle it?

Answer 1

Each service may have its own retry policy.

API Gateway will not retry a failed invocation of a Lambda. If you want to handle a retry, this will have to be done in the client calling the API Gateway.

A 502 error is, as you suggested, returned by the API Gateway when it receives a malformed Lambda proxy response (see https://aws.amazon.com/premiumsupport/knowledge-center/malformed-502-api-gateway/).