How to import JSON file to ElasticSearch 6.x?

Question

I got a brand new empty installation of Elastic Search 6.1.1 running on port 9200 of my CentOS7 host. I have this "es.json" file with data sample ready to be inserted in the ES.

I also have a log file in which each line is a json chunk. I could use both to populate my ES database.

How do I insert this and other data to ES?

I didnt find good explanations on the documentation and on the internet. for some reason the docs does not make it very clear

Answer 1

Use the --data-binary flag in curl to bulk import from a JSON file.

curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/{index}/{type}/_bulk?pretty' --data-binary @es.json

Data can be posted to one of the endpoints - /_bulk, /{index}/_bulk or {index}/{type}/_bulk. When {index} or {index}/{type} is provided, they will be used by default on bulk items that don’t provide them explicitly.

Content-Type: application/x-ndjson stands for Newline delimited JSON.

Before importing the JSON file, you might want to define mappings yourself or let Elasticsearch generate mappings dynamically during import. If you don't want Elasticsearch to generate mappings dynamically during import, refer to this doc to define mappings yourself.

References:

• https://www.elastic.co/guide/en/elasticsearch/reference/6.1/docs-bulk.html
• https://www.elastic.co/guide/en/kibana/6.1/tutorial-load-dataset.html

Answer 2

You can use elasticsearch_loader for loading json files into elasticsearch (2.X, 5.X, 6.X).

You can download it with pip:

pip install elasticsearch-loader

And then you will be able to load json files into elasticsearch by issuing:

elasticsearch_loader --index incidents --type incident json file1.json file2.json

Disclaimer: I'm the author of elasticsearch_loader