Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Querying a GSI immediately after DynamoDB streams event received - is it reliable?

From the below quote, I struggle to understand whether the propagation of a transaction to global secondary indexes (GSIs), streams, and backups happen in parallel, or is sequential.

Once a transaction completes, the changes made within that transaction are propagated to global secondary indexes (GSIs), streams, and backups.

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/transaction-apis.html

Hence my question: Is a dynamodb stream event raised only after all GSIs are rebuilt(replicated) or such a statement is false? In other words, can I be sure that I can find a projection for an item in some GSI if I query the GSI with the corresponding keys, immediately after I receive a stream event for its INSERT

To give some context - I am working on a concept of a procedure that can span across multiple lambdas, and each lambda is doing only one transaction. Prior to starting the procedure I know exactly how many transactions I should expect, and I am counting them with the help of dynamo streams and atomic counters, this way I can determine when a procedure has finished(edit Please do not redirect me to using step functions, I know their great potential but my stack will explode :))

However, if the above quote from AWS does not guarantee GSI replication prior to issuing the dynamo stream event, I am a bit screwed, because I can not immediately start other procedures that rely on the data inserted but want to query for it via a GSI...

Thanks!

edit: After helpful discussions and some more digging, I found this aws article that provides me with further understanding, particularly the Best practices for working with DynamoDB Streams section

like image 339
akrsmv Avatar asked Oct 23 '25 12:10

akrsmv


1 Answers

Stream events and GSI replication occur simultaneously. There is no guarantee one will occur before the other. With properly configured GSI's that are not experiencing any back pressure from hot partitions replication will occur in less than 10ms. Stream event processing typically occurs in less than 1 second. With a well designed data model it is highly likely that GSI replication will complete before a Stream event is processed, but there is no guarantee that this will be the case.

like image 172
Rick Houlihan Avatar answered Oct 26 '25 07:10

Rick Houlihan