Reduce DynamoDB latency from Java Lambda

Question

How can I reduce the latency that I'm currently seeing between Lambda & DynamoDB?

It's a Java lambda and it's taking 4s to perform each of 2 DynamoDB operations, using the AWS-supplied SDK. I've heard that these typically complete within <20ms, but for me it's 3 orders of magnitude higher. These long operations are (a) creating the DynamoDB object and (b) performing a table scan on a completely empty table (code below).

What should I do to reduce the latency?

Things I've tried

• Both the lambda and DynamoDB are in the same region (eu-west-1).
• The table has 5 RCU & WCU. Increasing these doesn't help.
• The maximum memory used by the lambda is 92MB. If I allocate the minimum 128MB then it times out after 15s. Increasing the memory to 512MB achieves the timings of 4s per call and increasing it again to 1GB reduces it to 2s per call. However, this is an absurd amount of memory for a trivial lambda and still leaves me with >200x the expected latency.
• Table metrics show the table scan time is between 12ms and 15ms. This is what I was expecting. Even with the overhead of establishing a network connection, I'm still expecting a few 10s of milliseconds, not several seconds.
• I'm triggering the lambda using the test function in the AWS console. I've also tried triggering via API Gateway (which is what I'll be doing eventually) with the same results.
• I've tried calling the lambda several times in quick succession (to reduce the likelihood that I'm suffering setup costs). It didn't help.
• Logging shows all other parts of the lambda running very quickly (milliseconds).

Code snippets

Creating the DynamoDB objects

log("Creating AmazonDynamoDB");
AmazonDynamoDB db = AmazonDynamoDBClientBuilder                        
  .standard()
  .withRegion(Regions.EU_WEST_1)
  .build();

log("Creating DynamoDBMapper");
DynamoDBMapper mapper = new DynamoDBMapper(db);

Performing the scan

log("Scanning table");
List<MyItem> items = dbMapper.scan(MyTable.class, new DynamoDBScanExpression());
for (MyItem item : items) {
        // Irrelevant - there aren't any
}
log("Table scan complete");

Sample logs

And here are the logs from a run.

20:07:41 START RequestId: 9d436db7-5d32-11e8-8555-8564d2094ccc Version: $LATEST
20:07:41 Received request: APIGatewayRequest(path=/data/foo, httpMethod=POST, body=)
20:07:41 Creating AmazonDynamoDB
20:07:45 Creating DynamoDBMapper
20:07:45 Creating DataHandler
20:07:45 Handling request
20:07:45 Scanning table
20:07:49 Table scan complete
20:07:49 Request handled - response object: []
20:07:49 APIGatewayResponse(isBase64Encoded=false, statusCode=200, body=[], headers={})
20:07:49 END RequestId: 9d436db7-5d32-11e8-8555-8564d2094ccc
20:07:49 REPORT RequestId: 9d436db7-5d32-11e8-8555-8564d2094ccc Duration: 8256.47 ms Billed Duration: 8300 ms Memory Size: 512 MB Max Memory Used: 85 MB

Answer 1

According to this post from an AWS employee on the AWS forums, construction of the AmazonDynamoDB object is expensive. Moving construction (back) into the static initializer combined with a little extra memory (=CPU) allocation basically sorts the problem.

Data from the logs still shows that each of the 2 slow steps identified above takes about half the time. Therefore, presumably both construction and first use of the AmazonDynamoDB objects are slow.

Obviously this doesn't help with the first request which still takes the same time as in the question. However, once the lambda is warmed, subsequent requests take ~15ms (well below the minimum billing threshold of 100ms). Addressing the first request problem is well understood - e.g. by using CloudWatch Events to schedule a regular call to the lambda to keep it warm.

Edit in 2020: You can also use Provisioned Currency to deal with the cold start problem.

Answer 2

(isn't a answer, but i hope it helps someone else) I have made the updates posted here and i besides them, i had to made a "dummy" query operation on dynamoDb (to open a connection with it), just in case of if help someone else, my code is as follows:

class MyFunctionHandler : RequestHandler<Map<String, Any>, ApiGatewayResponse> {

//var dbClient: AmazonDynamoDB = AmazonDynamoDBClientBuilder.defaultClient()
var dbClient: AmazonDynamoDB = AmazonDynamoDBClientBuilder
        .standard().withRegion("sa-east-1").build()

override fun handleRequest(input: Map<String, Any>, context: Context): ApiGatewayResponse {
    LOG.info("received input: $input")

    input["wakeup"]?.let {

        if (it == true) {

            with(EmpresaRepository(dbClient)) {
                LOG.info("### Connection was not stablished at this point")

                someDynamoQueryHere("dummyParameter")

                LOG.info("### The Connection was opened and will keep alived for 1 minute")
            }

            return buildResponseForWakeUpReq(input)
        }
    }

    val param = input["queryStringParameters"]?.toString()
...

The subsequent operations that will open the dynamoDb connection will occurs in terms of milliseconds!