Write more than 25 items using BatchWriteItemEnhancedRequest Dynamodb JAVA SDK 2

Question

I have an List items to be inserted into the DynamoDb collection. The size of the list may vary from 100 to 10k. I looking for an optimised way to Batch Write all the items using the BatchWriteItemEnhancedRequest (JAVA SDK2). What is the best way to add the items into the WriteBatch builder and then write the request using BatchWriteItemEnhancedRequest?

My Current Code:

   WriteBatch.Builder<T> builder = BatchWriteItemEnhancedRequest.builder().writeBatches(builder.build()).build();

   items.forEach(item -> { builder.addPutItem(item); });

   BatchWriteItemEnhancedRequest bwr = BatchWriteItemEnhancedRequest.builder().writeBatches(builder.build()).build()

   BatchWriteResult batchWriteResult =
            DynamoDB.enhancedClient().batchWriteItem(getBatchWriteItemEnhancedRequest(builder));

    do {
        // Check for unprocessed keys which could happen if you exceed
        // provisioned throughput
        List<T> unprocessedItems = batchWriteResult.unprocessedPutItemsForTable(getTable());

        if (unprocessedItems.size() != 0) {
            unprocessedItems.forEach(unprocessedItem -> {
                builder.addPutItem(unprocessedItem);
            });
            batchWriteResult = DynamoDB.enhancedClient().batchWriteItem(getBatchWriteItemEnhancedRequest(builder));
        }

    } while (batchWriteResult.unprocessedPutItemsForTable(getTable()).size() > 0);

Looking for a batching logic and a more better way to execute the BatchWriteItemEnhancedRequest.

Answer 1

I came up with a utility class to deal with that. Their batches of batches approach in v2 is overly complex for most use cases, especially when we're still limited to 25 items overall.

public class DynamoDbUtil {
    private static final int MAX_DYNAMODB_BATCH_SIZE = 25;  // AWS blows chunks if you try to include more than 25 items in a batch or sub-batch

    /**
     * Writes the list of items to the specified DynamoDB table.
     */
    public static <T> void batchWrite(Class<T> itemType, List<T> items, DynamoDbEnhancedClient client, DynamoDbTable<T> table) {
        Stream<List<T>> chunksOfItems = Lists.partition(items, MAX_DYNAMODB_BATCH_SIZE);
        chunksOfItems.forEach(chunkOfItems -> {
            List<T> unprocessedItems = batchWriteImpl(itemType, chunkOfItems, client, table);
            while (!unprocessedItems.isEmpty()) {
                // some failed (provisioning problems, etc.), so write those again
                unprocessedItems = batchWriteImpl(itemType, unprocessedItems, client, table);
            }
        });
    }

    /**
     * Writes a single batch of (at most) 25 items to DynamoDB.
     * Note that the overall limit of items in a batch is 25, so you can't have nested batches
     * of 25 each that would exceed that overall limit.
     *
     * @return those items that couldn't be written due to provisioning issues, etc., but were otherwise valid
     */
    private static <T> List<T> batchWriteImpl(Class<T> itemType, List<T> chunkOfItems, DynamoDbEnhancedClient client, DynamoDbTable<T> table) {
        WriteBatch.Builder<T> subBatchBuilder = WriteBatch.builder(itemType).mappedTableResource(table);
        chunkOfItems.forEach(subBatchBuilder::addPutItem);

        BatchWriteItemEnhancedRequest.Builder overallBatchBuilder = BatchWriteItemEnhancedRequest.builder();
        overallBatchBuilder.addWriteBatch(subBatchBuilder.build());

        return client.batchWriteItem(overallBatchBuilder.build()).unprocessedPutItemsForTable(table);
    }
}