How to do database schema migrations as part of AWS CDK deploy?

Question

We are running a web application on AWS EC2 that connects to a MariaDB running on AWS RDS. We have AWS CDK scripts set up for creating a new infrastructure including everything from our own services (the web application itself, background workers, RabbitMQ, RDS DB) and AWS services such as ALB, Security groups, DNS, S3 etc. Everything works fine, i.e. an operational infra, all services running, monitoring and autoscaling are set up when I run cdk deploy.

However, at times we need to upgrade MariaDB schema. In old infra with no autoscaling it was simply done by setting the one EC2 instance to maintenance mode, deploying new software and running special manage commands to initiate DB schema upgrades using Alembic. The problem with running just cdk deploy is that the new software versions of the web application may be incompatible with the DB schema until the DB schema has been upgraded. The software that is running should always correspond with the DB schema, i.e. there should not be any old instances running when the DB schema has been upgraded.

1. How to do cdk deploy in a controlled manner when there is a DB schema upgrade to be done so that there will not be instances running on an older software version that are incompatible with the new DB schema?

2. How to do DB schema upgrades as a part of cdk deploy?

Regarding question 2: We have considered having a special version of the web application which would not be a part of autoscaling or open to the Internet, i.e. it would be used only for running special manager commands such as DB schema upgrades. Is this unnecessary or does AWS CDK have some mechanisms to do controlled deployment with relational DB schema upgrades?

Also, we plan to use AWS CDK as part of CI/CD, i.e. run it with Jenkins to deploy software to target environment. Previously we have been orchestrating all this using Salt from Jenkins.

NB: I have seen this question: How to perform database schema migrations after deploying with AWS CDK? but this different since this is more about software compatibility and I think the answers there missed the actual question.

Answer 1

I think it is always a good pattern with CloudFormation/CDK to separate the persistent resources from app resources into separate Stacks (data/network/app). This will ensure everything in dependent stacks is fully updated before any downstream stacks are updated. Then I think putting the schema update code in a lambda that is called via a CustomResource in the data stack will allow CDK to perform the schema update before app code is updated.