Best way to synchronize Elasticsearch with Mysql

Question

I am using elasticsearch in my spring web mvc project (spring-data-elasticsearch) and to synchronize with database (MySQL).

I am indexing the document from my app, if any new entity going to add in db tables at the same time, from service layer, I request to index this document to elasticsearch also.

Both db tables and elasticsearch index have same data and to delete and update operation on I am using same concept, performing the change operation on elasticsearch and db table, it is working fine.

Now I want to know what will be the disadvantages of this approach.

Or is there any best way to make our elasticsearch index up to date from db. I used logstash but what about the deleted entities

Answer 1

The disadvantage of Synchronous indexation is there is no retry if there is an error while creating index data.

At your place i will create a cronjob/batch ( for trigger it depends how much data are updated and how important is the update of index ) and this job will have execution status with logs

you will have the clear idea about your index and no missing data

And for indexes you can a FULL index mode & an UPDATE indexes mode ( you should add an update date on your tables )

Indexing strategy you have two phases and you can choose TWO_PHASES : you need a master & slave ==> while executing indexing on master the slave will respond to requests and when the indexing is over you synchronize DIRECT_MODE : drop index & create new one