Sync MongoDB with ElasticSearch

Question

I want to sync my MongoDB data to ElasticSearch, I read a lot of posts talking about elasticsearch river plugin and mongo connector, but all of them are deprecated for mongo 4 and elasticsearch 7!

As logstash is a proprietary software I would like to use it to sync both... Anyone knows how it's possible to do it?

Answer 1

You may sync MongoDB and Elasticsearch with Logstash; syncing is, in fact, one of the major applications of Logstash. After installing Logstash, all that you need to do is specify a pipeline for your use case: one or more input sources (MongoDB in your case) and one or more output sinks (Elasticsearch in your case), put as a config file (example follows) inside Logstash's pipeline directory; Logstash takes care of the rest.

Logstash officially provides plugins for a lot of commonly used data sources and sinks; those plugins let you read data from and write data to various sources with just a few configuration. You just need to find the right plugin, install it, and configure it for your scenario. Logstash has an official output plugin for Elasticsearch and its configuration is pretty intuitive. Logstash, however, doesn't provide any input plugin for MongoDB. You need to find a third party one; this one seems to be pretty promising.

In the end, your pipeline may look something like the following:

input {
  mongodb {
    uri => 'mongodb://10.0.0.30/my-logs?ssl=true'
    placeholder_db_dir => '/opt/logstash-mongodb/'
    placeholder_db_name => 'logstash_sqlite.db'
    collection => 'events_'
    batch_size => 5000
  }
}
output {
  stdout {
    codec => rubydebug #outputs the same thing as elasticsearch in stdout to facilitate debugging
  }
  elasticsearch {
    hosts => "localhost:9200"
    index => "target_index"
    document_type => "document_type"
    document_id => "%{id}"
  }
}