How to read an individual column from Dynamo-Db without using Scan in Node-js?

Question

I have 4.5 millions of records in my Dynamo Db.

I want to read the the id of each record as a batchwise.

i am expecting something like offset and limit like how we can read in Mongo Db.

Is there any way suggestions without scan method in Node-JS.

I have done enough research i can only find scan method which buffers the complete records from Dynamo Db and the it starts scanning the records, which is not effective in performance basis.

Please do give me suggestion.

Answer 1

From my point of view, there's no problem doing scans because (according to the Scan doc):

• DynamoDB paginates the results from Scan operations

• You can use the ProjectionExpression parameter so that Scan only returns some of the attributes, rather than all of them

The default size for pages is 1MB, but you can also specify the max number of items per page with the Limit parameter.

So it's just basic pagination, the same thing MongoDB does with offset and limit.

Here is an example from the docs of how to perform Scan with the node.js SDK.

Now, if you want to get all the IDs as a batchwise, you could wrap the whole thing with a Promise and resolve when there's no LastEvaluatedKey.

Below a pseudo-code of what you could do :

const performScan = () => new Promise((resolve, reject) => {
    const docClient = new AWS.DynamoDB.DocumentClient();
    let params = {
        TableName:"YOUR_TABLE_NAME",
        ProjectionExpression: "id",
        Limit: 100 // only if you want something else that the default 1MB. 100 means 100 items
    };
    let items = [];

    var scanExecute = cb => {
        docClient.scan(params, (err,result) => {
            if(err) return reject(err);

            items = items.concat(result.Items);
            if(result.LastEvaluatedKey) {
                params.ExclusiveStartKey = result.LastEvaluatedKey;
                return scanExecute();
            } else {
                return err
                    ? reject(err)
                    : resolve(items);
            }
        });
    };
    scanExecute();
});

performScan().then(items => {
    // deal with it
});