How can I find out what is consuming my DynamoDb tables Read Capacity?

Question

We have a DynamoDB table that we thought we'd be able to turn off and delete. We shut down the callers to the web services that queried it (and can see on the web server metrics that the callers have dropped to zero), but the AWS console is still showing Read Capacity consumption greater than zero.

However, every other graph that concerns reads is showing no data: Get latency, Put latency, Query latency, Scan latency, Get records, Scan returned item count, and Query returned item count are all blank. On other tables that I know to be in use, these charts show some data > 0.

On other tables that I know not to be in use, the Read Capacity graph only shows the provisioned line, no consumed line.

This table is still being written to via a Lambda filtering and aggregating events from a Kinesis stream. I've reviewed the Lambda code and it doesn't specifically read anything from the table – does read capacity get consumed when the lambda updates or overwrites the value for an existing key?

Answer 1

I opened a ticket with AWS support and they were able to find the IP that was consuming the read capacity. They used an internal tool to query logs that are not available to customers. They also confirmed that these events do not get emitted to Cloudtrail logs, which only contain events related to the table, such as re-provisioning, queries about metrics, etc.

They also shared this nugget that's relevant to the question:

Q: Does read capacity get consumed when the lambda updates or overwrites the value for an existing key?

A: Yes, when you issue an update item operation, Dynamodb does a Read/Get operation first and then does a PutItem to insert/overwrite existing Item. This is expensive as it consumes both RCU and WCU. I did also verify that there are no UpdateItem operations being made on this table.

They also pointed me at more Cloudwatch metrics that shed some more light on what's going on with the table behind the scenes. Finding this through navigation with a link, you go to

• Cloudwatch service
• Metrics in the left bar
• All Metrics tab
• Scroll down to AWS Namespaces section (Custom Namespaces section is on top, if you have defined any custom metrics)
• Select DynamoDB
• Select Table Operation Metrics
  Metrics will be organized by table name. The one that was most helpful was Operation=Query, Metric Name=Returned Item Count.

So the only answer to my question is: Open an AWS Support ticket.