What is the best way to migrate an ElasticSearch cluster to new hardware?

Question

I have an elastic search cluster running on 6 underpowered nodes. I want to migrate it to a new set of 4 very heavy-duty nodes. I'm trying to figure out the best way to do so. Since I'm going from 6 nodes to 4 nodes, I can't just copy data files from the old cluster to the new one. It looks like the snapshot and restore function is the way to go, but I can't find a documented way to create a snapshot on one set of hardware and restore it into another set. Has anybody ever done this sort of hardware upgrade with ElasticSearch?

Answer 1

To use snapshot/restore, you have to have a common mount point on all of the servers (ie NFS). You have to add the repository on both clusters, then snapshot on one and restore on the other. The exact commands are pretty well documented here: http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html

Another way to migrate is to join the clusters together initially and wait for everything to be green. Then work through the process of decommissioning the old slow nodes: How to remove node from elasticsearch cluster on runtime without down time

The final way to migrate is the way that @Zouzias recommended which is to use a program to copy the data from one cluster to another. There is an open source node.js based project here: https://github.com/mallocator/Elasticsearch-Exporter that will do what you need without having to write code.

Another way to migrate is to use an API available in 5.x -- reindex from remote from remote:

POST _reindex
{
  "source": {
    "remote": {
      "host": "http://otherhost:9200",
      "username": "user",
      "password": "pass"
    },
    "index": "source"
  },
  "dest": {
    "index": "dest"
  }
}

But be sure to read the documentation at the link provided because you have to set the reindex.remote.whitelist in you elasticsearch.yml file.