Menu
Although the Lucene logic structure, I'm trying to make my nested fields to be highlighted when some search result is present in their content.
Here is the explanation from Elasticsearch documentation (mapping nested type`)
Internal Implementation
Internally, nested objects are indexed as additional documents, but, since they can be guaranteed to be indexed within the same "block", it allows for extremely fast joining with parent docs.
Those internal nested documents are automatically masked away when doing operations against the index (like searching with a match_all query), and they bubble out when using the nested query.
Because nested docs are always masked to the parent doc, the nested docs can never be accessed outside the scope of the nested query. For example stored fields can be enabled on fields inside nested objects, but there is no way of retrieving them, since stored fields are fetched outside of the nested query scope.
I have an Elasticsearch index containing a mapping like the following:
{
"my_documents": {
"dynamic_date_formats": [
"dd.MM.yyyy",
"yyyy-MM-dd",
"yyyy-MM-dd HH:mm:ss"
],
"index_analyzer": "Analyzer2_index",
"search_analyzer": "Analyzer2_search_decompound",
"_timestamp": {
"enabled": true
},
"properties": {
"identifier": {
"type": "string"
},
"description": {
"type": "multi_field",
"fields": {
"sort": {
"type": "string",
"index": "not_analyzed"
},
"description": {
"type": "string"
}
}
},
"files": {
"type": "nested",
"include_in_root": true,
"properties": {
"content": {
"type": "string",
"include_in_root": true
}
}
},
"and then some other": "normal string fields"
}
}
}
I'm trying to execute a query like this:
{
"size": 100,
"query": {
"bool": {
"should": [
{
"nested": {
"path": "files",
"query": {
"bool": {
"should": {
"match": {
"content": {
"query": "burpcontrol",
"minimum_should_match": "85%"
}
}
}
}
}
}
},
{
"match": {
"description": {
"query": "burpcontrol",
"minimum_should_match": "85%"
}
}
},
{
"match": {
"identifier": {
"query": "burpcontrol",
"minimum_should_match": "85%"
}
}
} ]
}
},
"highlight": {
"pre_tags": [
"<span style=\"background-color: yellow\">"
],
"post_tags": [
"</span>"
],
"order": "score",
"no_match_size": 100,
"fragment_size": 50,
"number_of_fragments": 3,
"require_field_match": true,
"fields": {
"files.content": {},
"description": {},
"identifier": {}
}
}
}
The problem I have are:
If I use "require_field_match": false I obtain that, even if highlighting doesn't work on nested fields, the search term is highlighted anyway in ALL the fields.
This is the solution I'm actually using, but the performances are horrible. For 50 documents my query needs 25secs. 100 documents about 50secs. 10 documents 5secs.
And if I remove the nested field from the highlighting everything works fast as light!
I would like to have a flattened version of my nested fields (so to store them as normal objects/fields. To do this I should specify
"files": { "type": "nested", "include_in_root": true, ...
but I don't know why, after reindexing, I cannot see any additional flattened field in the document root (while I was expecting something like "files.content":["content1", "content2", "..."]).
If it would work it would be instead possible to access (in the flattened field) the content of the nested field, and perform the highlighting on it.
686
You can search nested fields using dot notation that includes the complete path, such as obj1.name . Multi-level nesting is automatically supported, and detected, resulting in an inner nested query to automatically match the relevant nesting level, rather than root, if it exists within another nested query.
The nested type is a specialised version of the object data type that allows arrays of objects to be indexed in a way that they can be queried independently of each other.
When a packed class contains an instance field that is a packed type, the data for that field is packed directly into the containing class. The field is known as a nested field .
ElasticSearch's highlighting feature gives us the ability to highlight a portion of text in a document that matches the search term. Highlighting is very useful and one of these uses is to make it easier to find a term in long texts, which can help the user in his search.
There are a number of things you can do here, with a parent/child relationship. I'll go over a few, and hopefully that will lead you in the right direction; it will still take lots of testing to figure out whether this solution is going to be more performant for you. Also, I left out a few of the details of your setup, for clarity. Please forgive the long post.
I set up a parent/child mapping as follows:
DELETE /test_index
PUT /test_index
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0
},
"mappings": {
"parent_doc": {
"properties": {
"identifier": {
"type": "string"
},
"description": {
"type": "string"
}
}
},
"child_doc": {
"_parent": {
"type": "parent_doc"
},
"properties": {
"content": {
"type": "string"
}
}
}
}
}
Then added some test docs:
POST /test_index/_bulk
{"index":{"_index":"test_index","_type":"parent_doc","_id":1}}
{"identifier": "first", "description":"some special text"}
{"index":{"_index":"test_index","_type":"child_doc","_parent":1}}
{"content":"text that is special"}
{"index":{"_index":"test_index","_type":"child_doc","_parent":1}}
{"content":"text that is not"}
{"index":{"_index":"test_index","_type":"parent_doc","_id":2}}
{"identifier": "second", "description":"some different text"}
{"index":{"_index":"test_index","_type":"child_doc","_parent":2}}
{"content":"different child text, but special"}
{"index":{"_index":"test_index","_type":"parent_doc","_id":3}}
{"identifier": "third", "description":"we don't want this parent"}
{"index":{"_index":"test_index","_type":"child_doc","_parent":3}}
{"content":"or this child"}
If I'm understanding your specs correctly, we would want a query for "special" to return every one of these documents except the last two (correct me if I'm wrong). We want docs that match the text, have a child that matches the text, or have a parent that matches the text.
We can get back parents that match the query like this:
POST /test_index/parent_doc/_search
{
"query": {
"match": {
"description": "special"
}
},
"highlight": {
"fields": {
"description": {},
"identifier": {}
}
}
}
...
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1.1263815,
"hits": [
{
"_index": "test_index",
"_type": "parent_doc",
"_id": "1",
"_score": 1.1263815,
"_source": {
"identifier": "first",
"description": "some special text"
},
"highlight": {
"description": [
"some <em>special</em> text"
]
}
}
]
}
}
And we can get back children that match the query like this:
POST /test_index/child_doc/_search
{
"query": {
"match": {
"content": "special"
}
},
"highlight": {
"fields": {
"content": {}
}
}
}
...
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 0.92364895,
"hits": [
{
"_index": "test_index",
"_type": "child_doc",
"_id": "geUFenxITZSL7epvB568uA",
"_score": 0.92364895,
"_source": {
"content": "text that is special"
},
"highlight": {
"content": [
"text that is <em>special</em>"
]
}
},
{
"_index": "test_index",
"_type": "child_doc",
"_id": "IMHXhM3VRsCLGkshx52uAQ",
"_score": 0.80819285,
"_source": {
"content": "different child text, but special"
},
"highlight": {
"content": [
"different child text, but <em>special</em>"
]
}
}
]
}
}
We can get back parents that match the text and children that match the text like this:
POST /test_index/parent_doc,child_doc/_search
{
"query": {
"multi_match": {
"query": "special",
"fields": ["description", "content"]
}
},
"highlight": {
"fields": {
"description": {},
"identifier": {},
"content": {}
}
}
}
...
{
"took": 3,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"hits": {
"total": 3,
"max_score": 1.1263815,
"hits": [
{
"_index": "test_index",
"_type": "parent_doc",
"_id": "1",
"_score": 1.1263815,
"_source": {
"identifier": "first",
"description": "some special text"
},
"highlight": {
"description": [
"some <em>special</em> text"
]
}
},
{
"_index": "test_index",
"_type": "child_doc",
"_id": "geUFenxITZSL7epvB568uA",
"_score": 0.75740534,
"_source": {
"content": "text that is special"
},
"highlight": {
"content": [
"text that is <em>special</em>"
]
}
},
{
"_index": "test_index",
"_type": "child_doc",
"_id": "IMHXhM3VRsCLGkshx52uAQ",
"_score": 0.6627297,
"_source": {
"content": "different child text, but special"
},
"highlight": {
"content": [
"different child text, but <em>special</em>"
]
}
}
]
}
}
However, to get back all the docs related to this query, we need to use a bool query:
POST /test_index/parent_doc,child_doc/_search
{
"query": {
"bool": {
"should": [
{
"multi_match": {
"query": "special",
"fields": [
"description",
"content"
]
}
},
{
"has_child": {
"type": "child_doc",
"query": {
"match": {
"content": "special"
}
}
}
},
{
"has_parent": {
"type": "parent_doc",
"query": {
"match": {
"description": "special"
}
}
}
}
]
}
},
"highlight": {
"fields": {
"description": {},
"identifier": {},
"content": {}
}
},
"fields": ["_parent", "_source"]
}
...
{
"took": 5,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"hits": {
"total": 5,
"max_score": 0.8866254,
"hits": [
{
"_index": "test_index",
"_type": "parent_doc",
"_id": "1",
"_score": 0.8866254,
"_source": {
"identifier": "first",
"description": "some special text"
},
"highlight": {
"description": [
"some <em>special</em> text"
]
}
},
{
"_index": "test_index",
"_type": "child_doc",
"_id": "geUFenxITZSL7epvB568uA",
"_score": 0.67829096,
"_source": {
"content": "text that is special"
},
"fields": {
"_parent": "1"
},
"highlight": {
"content": [
"text that is <em>special</em>"
]
}
},
{
"_index": "test_index",
"_type": "child_doc",
"_id": "IMHXhM3VRsCLGkshx52uAQ",
"_score": 0.18709806,
"_source": {
"content": "different child text, but special"
},
"fields": {
"_parent": "2"
},
"highlight": {
"content": [
"different child text, but <em>special</em>"
]
}
},
{
"_index": "test_index",
"_type": "child_doc",
"_id": "NiwsP2VEQBKjqu1M4AdjCg",
"_score": 0.12531912,
"_source": {
"content": "text that is not"
},
"fields": {
"_parent": "1"
}
},
{
"_index": "test_index",
"_type": "parent_doc",
"_id": "2",
"_score": 0.12531912,
"_source": {
"identifier": "second",
"description": "some different text"
}
}
]
}
}
(I included the "_parent" field to make it easier to see why docs were included in the results, as shown here).
Let me know if this helps.
Here is the code I used:
http://sense.qbox.io/gist/d69a4d6531dc063faa4b4e094cff2a472a73c5a6
57
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
