Vector based search in solr

Question

I am trying to implement dense vector based search in solr (currently using version 8.5.2). My requirement is

1. to store a dense vector representation for each document in solr in a field called vectorForm.
2. Now when a user issues some query, I am converting that query to some dense vector representation as well and now I want to get top 100 documents from solr that have highest dotProduct value between the query vector representation and vectorForm field (stored for each document above) in solr.

A few questions that I had around this are

1. What field type should be used to define the vectorForm field (does docValues with multiValued integers work best here)?
2. How do I efficiently do the above vector based retrieval? (keeping in mind that latency should be as low as possible)

I read that solr has dotProduct and cosinSimilarity functions but not able to understand how to use it here in my case, if thats the solution then any link towards an example implementation will help.

Any help or guidance will be a huge help for me.

Answer 1

Neural Search has been released with Apache Solr 9.0.

The DenseVectorField gives the possibility of indexing and searching dense vectors of float elements, defining parameters such as the dimension of the dense vector to pass in, the similarity function to use, the knn algorithm to use, etc...

Currently, it is still necessary to produce the vectors externally and then push the obtained embeddings into Solr.

At query time you can use the k-nearest neighbors (knn) query parser that allows finding the k-nearest documents to the query vector according to indexed dense vectors in the given field.

Here is our End-to-End Vector Search Tutorial that can definitely help you understand how to leverage this new Solr feature to improve the user search experience https://sease.io/2023/01/apache-solr-neural-search-tutorial.html