Kubernetes rolling deployments and database migrations

Question

When processing a rolling update with database migrations, how does kubernetes handle this?

For an instance - I have an app that gets updated from app-v1 to app-v2, which includes a migration step to alter an existing table. So this would mean it requires me to run something like db:migrate for a rails app once deployed.

When a rolling deployment takes place on 3 replica set. It will deploy from one pod to another. Potentially allowing PODs that don't have the new version of the app to break.

Although this scenario is not something that happens very often. It's quite possible that it would. I would like to learn about the best/recommended approaches for this scenario.

Answer 1

One way to prevent an old version from breaking is to split a migration into multiple steps.

E.g. you want to rename a column in the database. Renaming the column directly would break old versions of the app. This can be split into multiple steps:

• Add a db migration that inserts the new column
• Change the app so that all writes go to the old and new column
• Run a task that copies all values from the old to the new column
• Change the app that it reads from the new column
• Add a migration that remove the old column

This is unfortunately quite a hassle, but prevents having a downtime with a maintenance page up.

Answer 2

Kubernetes does NOT natively handle rolling updates with db migrations. This is application specific, so the application developer will have to handle it. You may have to do the same things as you would do in a non-k8s setting.

Anyway, the way I would do it is:

• Scale your replicas to 1.
• Update the image.
• Scale your replicas to 3.

This is not fool-proof but doesn't involve code change. There is a small window between the db:migrate step and the actual server listening where a request will go to the older replica (which will terminate as soon as the new replica is ready). That request may or may not fail depending on whether the code block is directly related to the schema change.

If I didn't care much about downtime, then I would just use the recreate strategy.