Automatically offload dynamo table to cloud search domain

Question

I'm using Dynamo DB pretty heavily for a service I'm building. A new client request has come in that requires cloud search. I see that a cloud search domain can be created from a dynamo table via the AWS console.

My question is this:

Is there a way to automatically offload data from a dynamo table into a cloud search domain via the API or otherwise at a specified time interval?

I'd prefer this to manually offloading dynamo documents to cloudsearch. All help greatly appreciated!

Answer 1

Here are two ideas.

1. The official AWS way of searching DynamoDB data with CloudSearch

   This approach is described pretty thoroughly in the "Synchronizing a Search Domain with a DynamoDB Table" section of http://docs.aws.amazon.com/cloudsearch/latest/developerguide/searching-dynamodb-data.html.

   The downside is that it sounds like a huge pain: you have to either re-create new search domains or maintain an update table in order to sync, and you'd need a cron job or something to execute the script.

2. The AWS Lambdas way

   Use the newish Lambdas event processing service. It is pretty simple to set up an event stream based on Dynamo (see http://docs.aws.amazon.com/lambda/latest/dg/wt-ddb.html).

   Your Lambda would then submit a search document to CloudSearch based on the Dynamo event. For an example of submitting a document from a Lambda, see https://gist.github.com/fzakaria/4f93a8dbf483695fb7d5

   This approach is a lot nicer in my opinion as it would continuously update your search index without any involvement from you.