Loading data from Amazon dynamoDB to redshift

Question

We have a DynamoDB table in production that is being continuously updated,we want to load all the records from dynamoDB to redshift.

we tried using the copy command,but since new records are continuously being inserted in table the copy command runs forever.

We want to know what is the best way to load the data from live dynamodb to redshift.

Answer 1

Consider looking into a DynamoDB Streams based solution. Streams provides an ordered log of the data plane events transpiring on each DynamoDB partition (so events for each primary key are absolutely ordered). You can use Kinesis Client Library and DynamoDB Streams Kinesis Adapter to process the Stream to Redshift.

DynamoDB Streams is currently in preview, but should be generally available soon.