Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How can I retrieve all items from the paginated result of a DynamoDBMapper query()?

I am trying to query DynamoDB with help of DynamoDBMapper in Java with both hashKey and rangeKey. But I am not getting all results, it returns only some part of it. My code looks like:

queryDynamoDb() {
  Condition rangeKeyCondition = new Condition()
    .withComparisonOperator(ComparisonOperator.GT.toString())
    .withAttributeValueList(new AttributeValue().withS("0"));

    DynamoDBQueryExpression queryExpression = new DynamoDBQueryExpression(
            new AttributeValue().withS(prefKey));

    queryExpression.setRangeKeyCondition(rangeKeyCondition);

    List<MyObj> myobjs = mapper.query(MyObj.class, queryExpression);
    return myobjs;
}

MyObj is properly annotated with DynamoDB annotations. So I am able to save the objects, but retrieval returns a partial result only.

The documentation of query within DynamoDBMapper says:

The query method returns the "lazy-loaded" collection. That is, initially it returns only one page of results. It makes a service call for the next page when needed.

Now the question is, how to tell the mapper to make a service call or that a page is needed, so it loads all pages (effectively all entries)?

like image 249
GG. Avatar asked May 29 '12 17:05

GG.


People also ask

How pagination work in DynamoDB?

DynamoDB paginates the results from Query operations. With pagination, the Query results are divided into "pages" of data that are 1 MB in size (or less). An application can process the first page of results, then the second page, and so on. A single Query only returns a result set that fits within the 1 MB size limit.

What does DynamoDB query return?

In a Query operation, DynamoDB retrieves the items in sorted order, and then processes the items using KeyConditionExpression and any FilterExpression that might be present. Only then are the Query results sent back to the client. A Query operation always returns a result set.

What is PaginatedQueryList?

Class PaginatedQueryList<T>Implementation of the List interface that represents the results from a query in AWS DynamoDB. Paginated results are loaded on demand when the user executes an operation that requires them.

What is ExclusiveStartKey?

ExclusiveStartKey. The primary key of the first item that this operation will evaluate. Use the value that was returned for LastEvaluatedKey in the previous operation. The data type for ExclusiveStartKey must be String, Number, or Binary. No set data types are allowed.


1 Answers

The Java code snippet within the Amazon DynamoDB documentation for the DynamoDBMapper Class is a bit unfortunate here (though technologically correct), the AWS SDK for Java API documentation for Class DynamoDBMapper is (naturally) more precise in this regard, see method query():

public <T> PaginatedQueryList<T> query(Class<T> clazz,
                                       DynamoDBQueryExpression queryExpression)

So the returned type is actually a Class PaginatedQueryList:

Implementation of the List interface that represents the results from a query in AWS DynamoDB. Paginated results are loaded on demand when the user executes an operation that requires them. Some operations, such as size(), must fetch the entire list, but results are lazily fetched page by page when possible. [emphasize mine]

That is, you really do not need to explicitly load anything during normal usage, insofar it is implicitly taken care of by the lazy-loading implementation of PaginatedQueryList<T>; however, if so desired for whatever reason, you can trigger it by operations requiring access to the entire collection, with the explicitly mentioned size() method being one of them apparently.

like image 113
Steffen Opel Avatar answered Sep 18 '22 15:09

Steffen Opel