How can I trigger an AWS Lambda in response to CouchDB change events using only AWS?

Question

CouchDB provides a _changes endpoint, with the option to use long-lived HTTP connections to provide a stream of change events.

When a change event is produced, I'd like an AWS Lambda function to be triggered. CouchDB's continuous changes feed seems appropriate, but is it possible to use this as an event source for Lambda?

Typically, you might use API Gateway to invoke a Lambda, but via a HTTP call to the gateway endpoint, not as a HTTP consumer (CouchDB doesn't appear to provide a webhook-like mechanism).

This could be solved by using follow to listen for changes and manually invoke a Lambda, but can this be solved using AWS alone?

Answer 1

For the purposes of this answer I'm going to assume that "using AWS alone" excludes other AWS compute primitives like EC2 and Fargate. Which could be used to solve the continuous streaming problem.

The short answer is no, there are currently no way in Lambda alone to maintain a connection with a lifetime greater than 300 seconds. Although there are other ways to solve this problem if you are willing to ease the continuous connection constraint.

The first option would be to create a Lambda function that pulls the current change feed and stores the sequence number in a DynamoDB table. You could use a periodic CloudWatch Event to trigger this Lambda function each time calling the CouchDB endpoint with the since parameter using the ID stored in DynamoDB.

A second option would be to recursively call your Lambda function. Connect to the continuous event feed passing the since argument and keep the feed open until your Lambda function is about to time-out. Before the timeout grab the sequence ID of the last event and asynchronously invoke your Lambda function with that sequence ID as an input parameter, allowing your first function to complete. There may be a little bit of delay between the invocations but passing the sequence ID will ensure that you do not lose any events.

A third option would be to model the flow from the second option using Step Functions which are based on Lambda but provide a state machine around your Lambda invocations.

Another option that doesn't require as much coordination between the functions but trades off some additional setup would be to use ECS and Fargate. Fargate manages all the underlying compute infrastructure for an ECS cluster. You will need to provide container images and ECS configuration (I would recommend CloudFormation) but you'll ease the timeout constraints.