AWS Lambda: 504 error when returning large data set

Question

I have a Lambda function that takes several arrays (~6) and returns all possible combinations.

When there are a few hundred possible combinations, the Lambda is successful. However, when there are several thousand possibilities, I get a failure (response below).

Note that I am compressing/decompressing the JSON in both directions to make the payload as small as possible.

I am running this with max memory (1536) and a 20 second timeout.

Any thoughts on what is causing this?

{
  "state": "rejected",
  "reason": {
    "name": "StatusCodeError",
    "statusCode": 504,
    "message": "504 - [object Object]",
    "error": {
      "message": "Endpoint request timed out"
    },
    "options": {
      "uri": "https://blahblah/prod/getCombinations",
      "method": "POST",
      "timeout": 120000,
      "json": {...
      },
      "simple": true,
      "resolveWithFullResponse": false
    },
    "response": {
      "statusCode": 504,
      "body": {
        "message": "Endpoint request timed out"
      },
      "headers": {
        "content-type": "application/json",
        "content-length": "41",
        "connection": "close",
        "date": "Thu, 20 Aug 2015 20:39:53 GMT",
        "x-amzn-requestid": "965d3b8d-477b-11e5-99d6-4102846d4b1e",
        "x-cache": "Error from cloudfront",
        "via": "1.1 b1103856e287e98f322630821d3c6e5b.cloudfront.net (CloudFront)",
        "x-amz-cf-id": "Dhk7ylTq6RDE74smC8uF8ajms8rpU0fp2dnexn4_I3qIXgvrrsg48w=="
      },
      "request": {
        "uri": {
          "protocol": "https:",
          "slashes": true,
          "auth": null,
          "host": "blahblah",
          "port": 443,
          "hostname": "blahblah",
          "hash": null,
          "search": null,
          "query": null,
          "pathname": "/prod/getCombinations",
          "path": "/prod/getCombinations",
          "href": "https://blahblah"
        },
        "method": "POST",
        "headers": {
          "accept": "application/json",
          "content-type": "application/json",
          "content-length": 10201
        }
      }
    }
  }
}

Answer 1

You can read the Amazon CloudFront 504 response "Endpoint request timed out" literally, i.e. a 504 Gateway Timeout generally indicates that The server was acting as a gateway or proxy and did not receive a timely response from the upstream server.

CloudFront complies to this standard, as explicated in Customizing Error Responses for example:

Server errors indicate a problem with the origin server. For example, the HTTP server is busy or unavailable. When a server error occurs, either your origin server returns an HTTP status code in the 500 range to CloudFront, or CloudFront doesn't get a response from your origin server for a certain period of time and assumes a 504 status code (gateway timeout). [emphasis mine]

As Michael thankfully pointed out, CloudFront's own request timeout for custom origins is 30 seconds:

DELETE, OPTIONS, PATCH, POST, and POST requests – If the origin doesn't respond within 30 seconds, CloudFront drops the connection and doesn't try again to contact the origin. The client can resubmit the request if necessary.

Given your AWS Lambda function has a shorter timeout of 20 seconds, I'd suspect your dataset to be too large already to be transmitted and processed via a POST request (presumably through Amazon API Gateway?!) within 20 seconds.

• If your dataset is larger than a 1.5GB Lambda function with a 20 seconds execution duration can handle, you would need to either slice your data into smaller chunks or migrate your computation off Lambda onto something more scalable - by chance the AWS Compute Blog just features a nice sample how to facilitate Amazon ECS for longer running tasks like this, see Chris Barclay's Better Together: Amazon ECS and AWS Lambda.