Menu
I am using elasticsearch in my spring web mvc project (spring-data-elasticsearch) and to synchronize with database (MySQL).
I am indexing the document from my app, if any new entity going to add in db tables at the same time, from service layer, I request to index this document to elasticsearch also.
Both db tables and elasticsearch index have same data and to delete and update operation on I am using same concept, performing the change operation on elasticsearch and db table, it is working fine.
Now I want to know what will be the disadvantages of this approach.
Or is there any best way to make our elasticsearch index up to date from db. I used logstash but what about the deleted entities
964
You can use the SQL Gateway from the ODBC Driver for Elasticsearch to query Elasticsearch data through a MySQL interface.
The main difference ElasticSearch from MySQl-search is that ES works faster when large amounts of data through indexing. The index contains ready-made sets of data with which you are operating further ES-filters. So if you search with ES, you haven't to do a direct request to the database, as in MySQL.
Because Elasticsearch is not a relational database, joins do not exist as a native functionality like in an SQL database. It focuses more on search efficiency as opposed to storage efficiency. The stored data is practically flattened out or denormalized to drive fast search use cases.
You can easily set up your Elasticsearch to MySQL connection using CRUD Queries with the following steps: 1 Step 1: Creating a Custom Cluster and Node 2 Step 2: Creating CRUD Functions 3 Step 3: Search the Elasticsearch Data More ...
In order to synchronize our data from MySQL with the user index in ElasticSearch, we're gonna be using a Scheduler in which we will implement the synchronization logic. First of all, we need to enable Spring's scheduled task execution capability by adding the EnableScheduling annotation :
For this blog we use Logstash with the JDBC input plugin to keep Elasticsearch synchronized with MySQL. Conceptually, Logstash’s JDBC input plugin runs a loop that periodically polls MySQL for records that were inserted or modified since the last iteration of this loop.
you need to create a join query to get the data from all tables and then insert it in elasticsearch. Response time is depend upon the number of data you have. Thank you for pointing that out :).
The disadvantage of Synchronous indexation is there is no retry if there is an error while creating index data.
At your place i will create a cronjob/batch ( for trigger it depends how much data are updated and how important is the update of index ) and this job will have execution status with logs
you will have the clear idea about your index and no missing data
And for indexes you can a FULL index mode & an UPDATE indexes mode ( you should add an update date on your tables )
Indexing strategy you have two phases and you can choose TWO_PHASES : you need a master & slave ==> while executing indexing on master the slave will respond to requests and when the indexing is over you synchronize DIRECT_MODE : drop index & create new one
110
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
