Random status code: 502 errors on AWS api gateway connected to lambda

Question

I have multiple lambdas exposed with api gateway using proxy integration. From time to time i'm getting strange errors with status code 502. There is nothing in lambda cloud watch logs. Below i posted api gateway logs for sample request:

(0cbbd9f5-f1bd-11e7-92c0-4d5d3b7d0380) Received response. Integration latency: 231 ms

(0cbbd9f5-f1bd-11e7-92c0-4d5d3b7d0380) Endpoint response body before transformations:
{
    "Message": "An error occurred and the request cannot be processed.",
    "Type": "Service"
}

(0cbbd9f5-f1bd-11e7-92c0-4d5d3b7d0380) Endpoint response headers: 
{
    Connection=keep-alive, 
    x-amzn-RequestId=0cbc9dee-f1bd-11e7-857b-91f7f814692c, 
    x-amzn-ErrorType=ServiceException, 
    Content-Length=86, 
    Date=Fri, 05 Jan 2018 02:06:32 GMT, 
    Content-Type=application/json
}

(0cbbd9f5-f1bd-11e7-92c0-4d5d3b7d0380) Execution failed due to configuration error: Malformed Lambda proxy response

(0cbbd9f5-f1bd-11e7-92c0-4d5d3b7d0380) Method completed with status: 502

Basically it seems that api gateway cannot reach lambda and call to lambda is returning:

(0cbbd9f5-f1bd-11e7-92c0-4d5d3b7d0380) Endpoint response body before transformations:
{
    "Message": "An error occurred and the request cannot be processed.",
    "Type": "Service"
}

Is there any one else experiencing those issues? Only possible fix from my side is to write retry mechanism but from my side it looks rather that i am missing some configuration or it's AWS failure which they should handle.

Answer 1

I'm listing here one possible reason...

When an AWS Lambda is configured to run in VPC. It takes one IP per execution from VPC.

And if VPC doesn't much free IPs then your lambda will fail silently :(

I've personally faced issues in regards to limited IP, increasing the IPs solved the issue.

Below text from this link

The subnets you specify should have sufficient available IP addresses to match the number of ENIs.

We also recommend that you specify at least one subnet in each Availability Zone in your Lambda function configuration. By specifying subnets in each of the Availability Zones, your Lambda function can run in another Availability Zone if one goes down or runs out of IP addresses.

Note

If your VPC does not have sufficient ENIs or subnet IPs, your Lambda function will not scale as requests increase, and you will see an increase in function failures. AWS Lambda currently does not log errors to CloudWatch Logs that are caused by insufficient ENIs or IP addresses. If you see an increase in errors without corresponding CloudWatch Logs, you can invoke the Lambda function synchronously to get the error responses (for example, test your Lambda function in the AWS Lambda console because the console invokes your Lambda function synchronously and displays errors).