Pagination in DynamoDB using Node.js?

Question

I've had a read through AWS's docs around pagination:

As their docs specify:

In a response, DynamoDB returns all the matching results within the scope of the Limit value. For example, if you issue a Query or a Scan request with a Limit value of 6 and without a filter expression, DynamoDB returns the first six items in the table that match the specified key conditions in the request (or just the first six items in the case of a Scan with no filter)

Which means that given I have a table called Questions with an attribute called difficulty(that can take any numeric value ranging from 0 to 2) I might end up with the following conundrum:

• A client makes a request, think GET /questions?difficulty=0&limit=3
• I forward that 3 to the DynamoDB query, which might return 0 items as the first 3 in the collection might not be of difficulty == 0
• I then have to perform a new query to fetch more questions that match that criteria without knowing I might return duplicates

How can I then paginate based on a query correctly? Something where I'll get as many results as I asked for whilst having the correct offset

Answer 1

Using async/await.

const getAllData = async (params) => { 

    console.log("Querying Table");
    let data = await docClient.query(params).promise();

    if(data['Items'].length > 0) {
        allData = [...allData, ...data['Items']];
    }

    if (data.LastEvaluatedKey) {
        params.ExclusiveStartKey = data.LastEvaluatedKey;
        return await getAllData(params);

    } else {
        return data;
    }
}

I am using a global variable allData to collect all the data.

Calling this function is enclosed within a try-catch

try {

        await getAllData(params);
        console.log("Processing Completed");

        // console.log(allData);

    } catch(error) {
        console.log(error);
    }

I am using this from within a Lambda and it works fine.

The article here really helped and guided me. Thanks.

Answer 2

Here is an example of how to iterate over a paginated result set from a DynamoDB scan (can be easily adapted for query as well) in Node.js.

You could save the LastEvaluatedKey state serverside and pass an identifier back to your client, which it would send with its next request and your server would pass that value as ExclusiveStartKey in the next request to DynamoDB.

const AWS = require('aws-sdk');
AWS.config.logger = console;

const dynamodb = new AWS.DynamoDB({ apiVersion: '2012-08-10' });

let val = 'some value';

let params = {
  TableName: "MyTable",
  ExpressionAttributeValues: {
    ':val': {
      S: val,
    },
  },
  Limit: 1000,
  FilterExpression: 'MyAttribute = :val',
  // ExclusiveStartKey: thisUsersScans[someRequestParamScanID]
};

dynamodb.scan(scanParams, function scanUntilDone(err, data) {
  if (err) {
    console.log(err, err.stack);
  } else {
    // do something with data

    if (data.LastEvaluatedKey) {
      params.ExclusiveStartKey = data.LastEvaluatedKey;

      dynamodb.scan(params, scanUntilDone);
    } else {
      // all results scanned. done!
      someCallback();
    }
  }
});