Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Is there a way to filter the AWS DynamoDb Stream before triggering lambdas?

So we have a couple of lambdas that listen to changes on the db, in every lambda we had to filter in the beginning because this particular lambda don't care about all changes just a particular one.

To be more explicit, we are applying event sourcing and lambdas are supposed to be event handlers. I want lambda A to be triggered ONLY when event A is inserted in the db rather than whenever an event is inserted! And same for lambda B and event B and so on.

In the time being we have a filter in the beginning of every lambda:

lambdaA handler:
const eventsToBeProcessed = events.filter(
   (event) => event.eventName === 'EventA'
);

Now that we have a good bunch of lambdas it doesnt make sense to trigger all of them when I need only to trigger one!

I'm not very experienced with AWS but I'm assuming the solution will either be:

  • Being able to filter on DynamoDb Triggers
  • Instead of making lambdas triggered by dynamodb, have a AWS service in between that's triggered on every change and it will know which lambda to trigger depending on what's the update

EDIT:

I was not very happy with the solution I marked as an answer, the main reason because the Single Point Of Failure in the design, also it will need to count on an SNS to do a publish/subscribe and SNS is not highly available and it can fail you if you are trying to have a highly available system (4 9s or so)

The solution I ended up adapting (for now at least) is that when I push my events into the event store, if I want to trigger a side effect, I will just push it into AWS' EventBridge (an Event bus with rules). The best part about the EventBridge is that you can set the rules you want, the bus you want, and then have (for example) Lambda A be invoked when the bus have Event A and so on.

like image 331
Ali_Nass Avatar asked Feb 03 '21 10:02

Ali_Nass


2 Answers

There is now a way to do that,

https://aws.amazon.com/blogs/compute/filtering-event-sources-for-aws-lambda-functions/

It's a new feature of lambda, it uses the same syntax as AWS Eventbdridge.

aws lambda create-event-source-mapping \
--function-name fleet-tire-pressure-evaluator \
--batch-size 100 \
--starting-position LATEST \
--event-source-arn YOUR_DB_STREAM_ARN \
--filter-criteria '{"Filters": [{"eventName": ["EventA"]}]}'

You can refer to this documentation to know the exact syntax

like image 61
Jérémy Caré Avatar answered Oct 18 '22 10:10

Jérémy Caré


I think it would make sense to have one Lambda function responding to additions to the DynamoDB stream, dispatching the events to the respective downstream functions.

                                         ------------
                                     -->  Function A
--------     ---------------------       ------------
 Stream  -->  Dispatcher Function                        
--------     ---------------------       ------------
                                     -->  Function B
                                         ------------
like image 28
Dennis Traub Avatar answered Oct 18 '22 11:10

Dennis Traub