How to migrate dynamodb data on major table change?

Question

During development structures and requirements change. Key and index settings need to be changed, that might break incremental table update. So my solution so far is to delete the table and recreate it from the cloudformation stack.

But how to solve this problem with a production deployment? Is it possible to automate dynamodb deployment as follows?

1. Create new table
2. Migrate data from old table to new table
3. Delete old table

Answer 1

Yes, it is perfectly possible to automate such a deployment structure. As long as you have code to create a table, it should be fairly straightforward to get all of the data from an old table, change the data, and then upload it all to a new table without any drops in up-time. If you write what language you would like to do such a thing in I can help a bit more.

I've done this before and I've added below a small generified code-sample on how you could do this in Java.

Java method for creating a table given the class of the object type stored in dynamo:

 /**
 * Creates a single table with its appropriate configuration (CreateTableRequest)
 */
public void createTable(Class tableClass) {
    DynamoDBMapper mapper = createMapper(); // you'll need your own function to do this.

    ProvisionedThroughput pt = new ProvisionedThroughput(1L, 1L);
    CreateTableRequest ctr = mapper.generateCreateTableRequest(tableClass);
    ctr.withProvisionedThroughput(new ProvisionedThroughput(1L, 1L));

    // Provision throughput and configure projection for secondary indices.
    if (ctr.getGlobalSecondaryIndexes() != null) {
        for (GlobalSecondaryIndex idx : ctr.getGlobalSecondaryIndexes()) {
            if (idx != null) {
                idx.withProvisionedThroughput(pt).withProjection(new Projection().withProjectionType("ALL"));
            }
        }
    }

    TableUtils.createTableIfNotExists(client, ctr);
}

Java method to delete table:

private static void deleteTable(String tableName) {
    AmazonDynamoDB client = AmazonDynamoDBClientBuilder.standard().build();
    DynamoDB dynamoDB = new DynamoDB(client);
    Table table = dynamoDB.getTable(tableName);
    try {
        System.out.println("Issuing DeleteTable request for " + tableName);
        table.delete();
        System.out.println("Waiting for " + tableName + " to be deleted...this may take a while...");
        table.waitForDelete();

    }
    catch (Exception e) {
        System.err.println("DeleteTable request failed for " + tableName);
        System.err.println(e.getMessage());
    }
}

I would scan the whole table and plop all of the content into a List and then map through that list, converting the objects into your new type, and then create a new table of that type but with a different name, push all of your new objects, and then delete the old table after switching any references you might have of the old table to the new one. Unfortunately this does mean that everything consuming your tables are going to need to be able to switch between your two staging tables.