How do I manage schema/mapping migrations/evolutions in Elasticsearch?

Question

Flyway is a very convenient schema migration/evolution tool in the RDBMS world. I'm looking for something similar for ES.

Even though ES is different from RDBMS and I get that, the whole point of a tool like Flyway is basically doing the same schema changes in multiple environments such as 5 developer environments and staging/production environments. Even if I go with the aliasing approach described in a blog post, I still need to do that create-new-index-then-load-data-into-it-then-update-alias cycle in each environment. What I'm looking for is an automated way of doing that.

I can't just ask each developer to run a particular script after they pull a particular commit. Nor do I want to remember to manually run scripts like that in staging and production environments after deploying the latest codebase. Especially when the person doing a deployment is not the one who wrote migration scripts. All that feels so 20 years ago.

The problem has been solved multiple times in the RDBMS world. There are multiple mature tools out there. Flyway is just one of them and is my favorite. But I can't find anything similar for ES. I googled half the Web for it. Either my googling skills are very poor or a tool like that doesn't exist.

What am I missing? Is there a tool I can't find? Or am I completely misunderstanding something about ES and a tool like that doesn't make sense because of something I don't yet understand?

Answer 1

For create-new-index-then-load-data-into-it-then-update-alias, what we do is:

1. We use templates for the mapping
2. And we use curator to create/update the index/alias automatically.

Still the curator has to be run periodically, but we run it in a cron job.

Answer 2

In 2020, there seems to be an easier approach: The reindex API. You only need to do

POST _reindex
{
  "source": {
    "index": "my-index-000001"
  },
  "dest": {
    "index": "my-new-index-000001"
  }
}

and the data gets re-indexed.

I am new to Elasticsearch so don't hesitate to point out where I can improve :)

Answer 3

You can do this to a certain extent with the elasticsearch-evolution tool, which describes itself as a "flyway for elastic": https://github.com/senacor/elasticsearch-evolution