ScannedCount is less than the Total number of records and result is zero in dynamoDB

Question

I am facing a weird issue on dynamoDB AWS - I am querying my table using AWS API Gateway - AWS Service Proxy and I get Count:0 results and the ScannedCount is approx 2500 records out of total 10000 records. Just to confirm I have required data in my table for which I am using Scan Operation on dynamoDB.

What I am not able to understand is why the ScannedCount is less than the complete table records. Is this suppose to happen

Answer 1

According to the DynamoDB documentation, ScannedCount is the number of items dynamodb has looked through for current request and Count is the number of items matched your filter:

Counting the Items in the Results

In addition to the items that match your criteria, the Query response contains the following elements:

• ScannedCount — the number of items that matched the key condition expression, before a filter expression (if present) was applied.
• Count — the number of items that remain, after a filter expression (if present) was applied.**

Note

If you do not use a filter expression, then ScannedCount and Count will have the same value.

If the size of the Query result set is larger than 1 MB, then ScannedCount and Count will represent only a partial count of the total items. You will need to perform multiple Query operations in order to retrieve all of the results (see Paginating the Results).

Each Query response will contain the ScannedCount and Count for the items that were processed by that particular Query request. To obtain grand totals for all of the Query requests, you could keep a running tally of both ScannedCount and Count.

So in your case the scan went through first 2500 records (ScannedCount is 2500) and there are no results matching your filter (Count is zero).

To scan the rest of the data in the table, you need to repeat the request with pagination parameters as described here:

A single Scan will only return a result set that fits within the 1 MB size limit. To determine whether there are more results, and to retrieve them one page at a time, applications should do the following:

• Examine the low-level Scan result:
  • If the result contains a LastEvaluatedKey element, proceed to step 2.
  • If there is not a LastEvaluatedKey in the result, then there are no more items to be retrieved.
• Construct a new Scan request, with the same parameters as the previous one—but this time, take the LastEvaluatedKey value from step 1 and use it as the ExclusiveStartKey parameter in the new Scan request.
• Run the new Scan request.
• Go to step 1.

Depending on the language, you can find a library that does the pagination for you, like boto2 high-level dynamodb client for python or "paginator" in boto3.