Using NEST to percolate

Question

I'm indexing my query as follows:

client.Index(new PercolatedQuery
{
    Id = "std_query",
    Query = new QueryContainer(new MatchQuery
    {
        Field = Infer.Field<LogEntryModel>(entry => entry.Message),
        Query = "just a text"
    })
}, d => d.Index(EsIndex));

client.Refresh(EsIndex);

Now, how do I use the percolator capabilities of ES to match an incoming document with this query? To say the NEST documentation is lacking in this area would be a huge understatement. I tried using client.Percolate call, but it's deprecated now and they advise to use the search api, but don't tell how to use it with percolator...

I'm using ES v5 and the same version of NEST lib.

Answer 1

There are plans to improve the documentation for 5.x once the GA release is out; I understand that documentation could be clearer in many places and any help in this area would be most appreciated :)

The documentation for the Percolate query is generated from the integration test for it. Pulling out all the pieces for an example here, using details from you other question. First, let's define the POCO models

public class LogEntryModel
{
    public string Message { get; set; }

    public DateTimeOffset Timestamp { get; set; }
}

public class PercolatedQuery
{
    public string Id { get; set; }

    public QueryContainer Query { get; set; }
}

We're going to fluently map all properties instead of using mapping attributes. The fluent mappings are the most powerful and can express all ways to map in Elasticsearch.

Now, create the connection settings and client to work with Elasticsearch.

var pool = new SingleNodeConnectionPool(new Uri($"http://localhost:9200"));
var logIndex = "log_entries";
var connectionSettings = new ConnectionSettings(pool)
    // infer mapping for logs
    .InferMappingFor<LogEntryModel>(m => m
        .IndexName(logIndex)
        .TypeName("log_entry")
    )
    // infer mapping for percolated queries
    .InferMappingFor<PercolatedQuery>(m => m
        .IndexName(logIndex)
        .TypeName("percolated_query")
    );

var client = new ElasticClient(connectionSettings);

We can specify the index name and type name to infer for our POCOs; that is, when NEST makes a request using LogEntryModel or PercolatedQuery as the generic type parameter in a request (e.g. the T in .Search<T>()), it will use the inferred index name and type name if they are no specified on the request.

Now, delete the index so that we can start from scratch

// delete the index if it already exists
if (client.IndexExists(logIndex).Exists)
    client.DeleteIndex(logIndex);

And create the index

client.CreateIndex(logIndex, c => c
    .Settings(s => s
        .NumberOfShards(1)
        .NumberOfReplicas(0)
    )
    .Mappings(m => m
        .Map<LogEntryModel>(mm => mm
            .AutoMap()
        )
        .Map<PercolatedQuery>(mm => mm
            .AutoMap()
            .Properties(p => p
                // map the query field as a percolator type
                .Percolator(pp => pp
                    .Name(n => n.Query)
                )
            )
        )
    )
);

The Query property on the PercolatedQuery is mapped as a percolator type. This is new in Elasticsearch 5.0. The mapping request looks like

{
  "settings": {
    "index.number_of_replicas": 0,
    "index.number_of_shards": 1
  },
  "mappings": {
    "log_entry": {
      "properties": {
        "message": {
          "fields": {
            "keyword": {
              "type": "keyword"
            }
          },
          "type": "text"
        },
        "timestamp": {
          "type": "date"
        }
      }
    },
    "percolated_query": {
      "properties": {
        "id": {
          "fields": {
            "keyword": {
              "type": "keyword"
            }
          },
          "type": "text"
        },
        "query": {
          "type": "percolator"
        }
      }
    }
  }
}

Now, we're ready to index the query

client.Index(new PercolatedQuery
{
    Id = "std_query",
    Query = new MatchQuery
    {
        Field = Infer.Field<LogEntryModel>(entry => entry.Message),
        Query = "just a text"
    }
}, d => d.Index(logIndex).Refresh(Refresh.WaitFor));

With the query indexed, Let's percolate a document

var logEntry = new LogEntryModel
{
    Timestamp = DateTimeOffset.UtcNow,
    Message = "some log message text"
};

// run percolator on the logEntry instance
var searchResponse = client.Search<PercolatedQuery>(s => s
    .Query(q => q
        .Percolate(p => p
            // field that contains the query
            .Field(f => f.Query)
            // details about the document to run the stored query against.
            // NOTE: This does not index the document, only runs percolation
            .DocumentType<LogEntryModel>()
            .Document(logEntry)
        )
    )
);

// outputs 1
Console.WriteLine(searchResponse.Documents.Count());

The percolated query with id "std_query" comes back in the searchResponse.Documents

{
  "took" : 117,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 0.2876821,
    "hits" : [
      {
        "_index" : "log_entries",
        "_type" : "percolated_query",
        "_id" : "std_query",
        "_score" : 0.2876821,
        "_source" : {
          "id" : "std_query",
          "query" : {
            "match" : {
              "message" : {
                "query" : "just a text"
              }
            }
          }
        }
      }
    ]
  }
}

This is an example of percolating a document instance. Percolation can also be run against already indexed documents

var searchResponse = client.Search<PercolatedQuery>(s => s
    .Query(q => q
        .Percolate(p => p
            // field that contains the query
            .Field(f => f.Query)
            // percolate an already indexed log entry
            .DocumentType<LogEntryModel>()
            .Id("log entry id")
            .Index<LogEntryModel>()
            .Type<LogEntryModel>()
        )
    )
);