Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Elasticsearch to index RDBMS data

These are three simple questions which was surprisingly hard to find definite answers.

  1. Does ElasticSearch support indexing data in RDBMS tables ( Oracle/SQLServer/Informix) out of the box?
  2. If yes, can you please point me to documentation on how to do it
  3. If not, what are alternate ways (plugins like Rivers - deprecated) with good reputation
like image 205
Ishan Hettiarachchi Avatar asked Jan 29 '16 05:01

Ishan Hettiarachchi


People also ask

Is Elasticsearch good for relational data?

Because Elasticsearch is not a relational database, joins do not exist as a native functionality like in an SQL database. It focuses more on search efficiency as opposed to storage efficiency. The stored data is practically flattened out or denormalized to drive fast search use cases.

How do I index a database in Elasticsearch?

To index content like HTML pages, catalogs, and other files, you can use Workplace Search content connectors, the Enterprise Search web crawler, or send data directly to Elasticsearch from your application using one of the Elastic language clients. The preferred way to index timestamped data is to use Elastic Agent.

Is Elasticsearch is RDBMS?

This can be achieved by adopting NoSQL rather than RDBMS for storing data. Elasticsearch is one such NoSQL distributed database. Elasticsearch relies on flexible data models to build and update visitors' profiles to meet the demanding workload and low latency required for real-time engagement.


1 Answers

I'm surprised there isn't any solid answer as yet for this. So here's the solution. Logstash directly gives us the ability to push data from a RDBMS into Elasticsearch.

Here's a link to a tutorial which tell you how to go about it. Briefly(all details in link 1), you simply need a JDBC driver for the relational database you'll be using (Postgres, MySQL etc) and make a config file specifying your input as the Relational Database and your output as Elasticsearch. You can also specify a cron which would allow you to keep updating one regular intervals.

  • Here's the article which mentions the configuration and gets you started (See Example 2): https://www.elastic.co/blog/logstash-jdbc-input-plugin

  • Here's the article which tells you how to configure the Cronjob as such: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html#_scheduling

like image 138
bholagabbar Avatar answered Sep 29 '22 14:09

bholagabbar