Document Similarity in ElasticSearch

Question

I want to calculate similarity between two documents indexed in elasticsearch. I know it can be done in lucene using term vectors. What is the direct way to do it?

I found that there is a similarity module doing exactly this: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-similarity.html

How do I integrate this in my system? I am using pyelasticsearch for calling elasticsearch commands, but I am open to use the REST api for similarity if needed.

Answer 1

I think the Elasticsearch documentation can easily be mis-interpreted.

Here "similarity" is not a comparison of documents or fields but rather a mechanism for scoring matching documents based on matching terms from the query.

The documentation states:

A similarity (scoring / ranking model) defines how matching documents are scored.

The similarity algorithms that Elasticsearch supports are probabilistic models based on term distribution in the corpus (index).

In regards to term vectors, this also can be mis-interpreted.

Here "term vectors" refer to statistics for the terms of a document that can easily be queried. It seems that any similarity measurements across term vectors would then have to be done in your application post-query. The documentation on term vectors state:

Returns information and statistics on terms in the fields of a particular document.

If you need a performant (fast) similarity metric over a very large corpus you might consider a low-rank embedding of your documents stored in an index for doing approximate nearest neighbor searches. After your KNN lookup, which greatly reduces the candidate set, you can do more costly metric calculations for ranking.

Here is an excellent resource for evaluation of approximate KNN solutions: https://github.com/erikbern/ann-benchmarks