How to index and store multiple languages in ElasticSearch

Question

I am trying to figure out gow to index the following in ES.

I have a lot of documents which are crawler from website with various language. Each document has a category such as Airport, restaurant, river, beach etc ., and it's language such as Arabic, English.. like

doc { language:"eng" , content :"something here" , category:"beach" }

doc { language:"vn" , content :"Xin chao" , category:"beach" }

I want to index and search documents with each languages;

I choose English options, and search with query " here " => RESUTLS

Should I :

1. Setup each Elastic Core ( per machine per language) for per language. JUST COPY ES to run :)

   Eg : create elasticsearch_ENGLISH, elastichsearch_VIETNAMESE

2. created each language with each index of ElasticSearch Eg: create indexs

/english/type/

/vietnames/type/ . When I search some query, I just search only index of language

OR do it some other way I am not aware of :) ?

Answer 1

Not sure I fully understood your concern.

If you need to search on the full cluster (I mean search in every language), you can't create one setup per language.

That said, you have many options:

• Create one index per language and have a mapping for each index/type.
• Use the _analyzer field to indicate the language for your document. See http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-analyzer-field.html
• Use a multifield with a different analyzer for each language. See https://www.elastic.co/guide/en/elasticsearch/reference/0.90/mapping-multi-field-type.html

It's not a full answer but some clues to help you...