Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Nested documents in Elasticsearch

Tags:

I'm writing an asset management application. It lets users store arbitrary asset attributes by adding an html control such as a text field, select menu, etc. to the asset. A JSON representation of the attribute then becomes part of the asset JSON document stored in couchdb. An asset has the following structure in couchdb:

{    "_id": "9399fb27448b1e5dfdca0181620418d4",    "_rev": "12-fa50eae8b50f745f9852e9fab30ef5d9",    "type": "asset",    "attributes": [        {            "id": "9399fb27448b1e5dfdca01816203d609",            "type": "text",            "heading": "Brand",            "data": "",            "requiredBySystem": true        },        {            "id": "9399fb27448b1e5dfdca01816203e68e",            "type": "userSelectMenu",            "heading": "Assigned To",            "data": "",            "requiredBySystem": true        },        {            "id": "9399fb27448b1e5dfdca01816203e9c9",            "type": "categories",            "heading": "Categories",            "data": [                "0d7e6233e5f48b4f55c5376bf00b1be5",                "0d7e6233e5f48b4f55c5376bf00d94cf"            ],            "requiredBySystem": true        },        {            "id": "9399fb27448b1e5dfdca01816207uy5a",            "type": "radio",            "heading": "Radio Buttons",            "data": [                {                    "text": "Button 1",                    "checked": false                },                {                    "text": "Button 2",                    "checked": true                }            ],            "requiredBySystem": true        },        {            "id": "9399fb27448b1e5dfdca01816205tgh6",            "type": "checkboxes",            "heading": "Checkboxes",            "data": [              {                  "text": "Box 1",                  "checked": false              },              {                  "text": "Box 2",                  "checked": true              }            ],            "requiredBySystem": true        },        {            "id": "9399fb27448b1e5dfdca0181620k81gt",            "type": "select",            "heading": "Select Menu",            "data": [                {                    "text": "Option 1",                    "checked": false                },                {                    "text": "Option 2",                    "checked": true                }            ],            "requiredBySystem": true        }    ] } 

I'm not sure if putting attributes in an array is the best way to allow searching for an asset based on an attribute value. Would it be better to attach the attribute directly to the asset as a property? I'm experimenting now in elasticsearch. If I try and store the document as is, elasticsearch returns an error:

"error" : "MapperParsingException[Failed to parse [attributes.data]]; nested: ElasticSearchIllegalArgumentException[unknown property [text]]; "

I"m using the following mapping:

"mappings" : {     "asset" : {       "properties" : {         "_id": {           "type" : "string",           "index" : "not_analyzed"         },         "_rev": {           "type" : "string",           "index" : "not_analyzed"         },         "type": {           "type" : "string",           "index" : "not_analyzed"         },         "attributes": {           "properties" : {             "id" : {               "type" : "string"             },             "type" : {               "type" : "string",               "index" : "not_analyzed"             },             "heading" : {               "type" : "string"             },             "data" : {               "type" : "string"             }           }         }       }     }   } 

Not sure where I'm going wrong here. Thanks for your help!

Troy

like image 302
Troy Avatar asked Mar 23 '13 03:03

Troy


People also ask

What is nested in elastic search?

The nested type is a specialised version of the object data type that allows arrays of objects to be indexed in a way that they can be queried independently of each other.

What is nested document?

Embedded document or nested documents are those types of documents which contain a document inside another document.

What is nested aggregation?

Nested aggregationeditA special single bucket aggregation that enables aggregating nested documents. For example, lets say we have an index of products, and each product holds the list of resellers - each having its own price for the product.

What is nested datatype?

Nested data types are structured data types for some common data patterns. Nested data types support structs, arrays, and maps. A struct is similar to a relational table. It groups object properties together.


1 Answers

The problem is arising from the way your document is structured. attribute.data is both a string/array of strings and a full, inner object. ES doesn't allow the "type" of a property to change.

Basically, you can't have this:

 "data": [   "0d7e6233e5f48b4f55c5376bf00b1be5",   "0d7e6233e5f48b4f55c5376bf00d94cf" ], 

and this:

 "data":[   {     "text":"Button 1",     "checked":false   },   {     "text":"Button 2",     "checked":true   } ], 

in the same document. The first instance of data tells ES that "data is an array of strings". But then the second instance of data says "Hey, I'm an object!", which is why ES is throwing an error.

You can sidestep this issue by explicitly declaring data as an object and setting enabled: false, but this is probably not the solution you want (since that just tells ES to store data as a text field, with no parsing.

The other option is to either restructure your data, or split data into it's document (e.g. Parent/Child mapping)

like image 51
Zach Avatar answered Oct 01 '22 23:10

Zach