How to use new secret created by key rotation from AWS Secrets Manager

Question

I have a Java application using MongoDB (or it could be any service like that). On start up, the app creates a singleton connection to database. To connect, I get the MongoDB from AWS Secrets Manager... and thus the application runs merrily ever after communicating with MongoDB.

My question is: What happens when AWS Secrets Manager rotates keys?

• How does my app come to 'know' that secret has been rotated.
• Do I have to synchronize the timing between Secrets Manager and my app?

e.g. rotation is set to 7 days. So I code in my app to refresh every 7 days... not good, as very hard to time precisely.

Another way could be, if my app faces authentication exception, just refresh password and make a new connection and retry app logic.

What is the industry standard?

Answer 1

How does my app come to 'know' that secret has been rotated?

-AWS Secrets Manager publishes the CloudTrail event - 'RotationSucceeded' when rotation succeeds and the cloudtrail event 'RotationFailed' when rotation fails. You can setup a cloudwatch rule on this cloudtrail event - https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/Create-CloudWatch-Events-CloudTrail-Rule.html

and have a SNS or a Lambda setup as the target for the rule and execute any logic you want after rotation succeeds

Answer 2

This is generally dealt with using one of two strategies, or in Secrets Manager speak, by using single user rotation or multi user rotation. Secrets Manager provides lambda implementations for both single and multi user rotations of MongoDB.

In single user rotation there is one DB user/password pair. During rotation the password is updated either using the original user/password or by fetching the master user creds and using those to update the password. In this case any connections established using the old creds would fail after rotation. To deal with this the application would use a connection manager that detected an authentication error (or all errors if necessary) and refreshes the secret before retrying. This is the strategy used by the Secrets Manager provided JDBC wrapper.

The other alternative (multi user rotation) is to read the user name from the original secret, and then, on the first rotation, create a clone of that user with a new password using the master user secret. After that rotation consists of alternating the secret user/password pair between the original and clone and updating the password. In this case the application only needs to refresh the secret once in the rotation interval. If it is using the old user/password pair, it will remain valid for two rotation intervals.

If you are using MongoDB on AWS (as apposed to DocumentDB with Mongo compatability), the easiest thing to do is spin up a temporary DocumentDB and use the Secrets Manager console to setup rotation on that. Then copy the Lambdas, roles and policies, and secrets used there for your Mongo application before tearing down the DocumentDB instance. If you are already using DocumentDB then as mentioned just use the SecretsManager console to set it up.