how to retrieve data using Dynamodb parallel scans using nodejs

Question

I am working on importing a dynamo table to S3 using nodejs. I have everything working fine but the entire copy is pretty slow as I have millions of records and the sequential scan is limited to 1MB, so I am looking in doing a parallel scan for dynamo dB using node.

But to do that, I need to create multiple threads and assign the parallel work to those threads in node. I would like to know if there is a recommended way of doing that, or do you suggest using Data Pipeline for importing data? How do you think I should go about it?

Answer 1

You can use the in-build parallel scan feature available in the Scan API. The entire will be divided into multiple segments and scan will be performed on the individual segments.

Parallel scan

Segment: 0,
TotalSegments: 5

Segment: For a parallel Scan request, Segment identifies an individual segment to be scanned by an application worker.

Segment IDs are zero-based, so the first segment is always 0. For example, if you want to use four application threads to scan a table or an index, then the first thread specifies a Segment value of 0, the second thread specifies 1, and so on.

The value for Segment must be greater than or equal to 0, and less than the value provided for TotalSegments.

If you provide Segment, you must also provide TotalSegments.

TotalSegments:- For a parallel Scan request, TotalSegments represents the total number of segments into which the Scan operation will be divided. The value of TotalSegments corresponds to the number of application workers that will perform the parallel scan. For example, if you want to use four application threads to scan a table or an index, specify a TotalSegments value of 4.