Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Function to scan AWS Dynamo DB recursively for Nodejs

So I need a recursive function in node.js for replacing this function call:

docClient.scan(params, callback)

More info see http://docs.aws.amazon.com/amazondynamodb/latest/gettingstartedguide/GettingStarted.NodeJs.04.html

like image 495
locropulenton Avatar asked Nov 25 '16 17:11

locropulenton


People also ask

How do I Scan a DynamoDB table in node JS?

DynamoDB service object. Create a JSON object containing the parameters needed to scan the table for items, which in this example includes the name of the table, the list of attribute values to return for each matching item, and an expression to filter the result set to find items containing a specified phrase.

Which is faster Scan or query in DynamoDB?

For faster response times, design your tables and indexes so that your applications can use Query instead of Scan . (For tables, you can also consider using the GetItem and BatchGetItem APIs.)

How can I make DynamoDB Scan faster?

Parallel Scan in DynamoDB Scans are generally speaking slow. To make that process faster, you can use a feature called "Parallel Scans" which divide the whole DynamoDB Table into Segments. A separate thread/worker then processes each Segment so N workers can work simultaneously to go through the whole keyspace faster.


1 Answers

Here is the recursive code to execute the scan until LastEvaluatedKey is available.

var AWS = require("aws-sdk");
var creds = new AWS.Credentials('akid', 'secret', 'session');

AWS.config.update({
    region: "us-west-2",
    endpoint: "http://localhost:8000",
    credentials : creds
});

var docClient = new AWS.DynamoDB.DocumentClient();

var params = {
    TableName: "Movies"
};

console.log("Scanning Movies table.");
docClient.scan(params, onScan);
var count = 0;

function onScan(err, data) {
    if (err) {
        console.error("Unable to scan the table. Error JSON:", JSON.stringify(err, null, 2));
    } else {
        // print all the movies
        console.log("Scan succeeded.");
        data.Items.forEach(function(movie) {
           console.log("Item :", ++count,JSON.stringify(movie));
        });

        // continue scanning if we have more movies
        if (typeof data.LastEvaluatedKey != "undefined") {
            console.log("Scanning for more...");
            params.ExclusiveStartKey = data.LastEvaluatedKey;
            docClient.scan(params, onScan);
        }
    }
}  
like image 59
notionquest Avatar answered Nov 04 '22 10:11

notionquest