Elasticsearch search fails in field with special character and wildcard

Question

I have a field in Elasticsearch with the value "PEI.H.02354.01.". When I search with querystring as

{  
   "query":{  
      "query_string":{  
         "query":"field:PEI.H.02354.01.",
         "default_operator":"AND"
      }
   }
}

then a result is returned, which is the correct behavior. But if I search with a wildcard, then no results are returned, e.g.

{  
   "query":{  
      "query_string":{  
         "query":"field:PEI.H.02354.01.*",
         "default_operator":"AND"
      }
   }
}

The field is of type string and analyzed. Below is the code that creates the index, including the analyzer and the mappings.

{
   "settings":{  
      "analysis":{  
         "analyzer":{  
            "number":{  
               "type":"custom",
               "tokenizer":"keyword",
               "filter":[  
                  "lowercase"
               ],
               "char_filter":[  
                  "number_filter"
               ]
            },
            "diacritical":{  
               "type":"custom",
               "tokenizer":"standard",
               "filter":[  
                  "standard",
                  "lowercase",
                  "asciifolding",
                  "nfd_normalizer"
               ]
            }
         },
         "filter":{  
            "nfd_normalizer":{  
               "type":"icu_normalizer",
               "name":"nfc"
            }
         },
         "char_filter":{  
            "number_filter":{  
               "type":"pattern_replace",
               "pattern":"[^\\d]+",
               "replacement":""
            }
         }
      }
   },
   "mappings":{  
      "testType":{  
         "_source":{  
            "enabled":false
         },
         "_all":{  
            "enabled":false
         },
         "_timestamp":{  
            "enabled":"true",
            "store":"yes"
         },
         "properties":{  
            "field":{  
               "store":"yes",
               "type":"string",
               "index":"analyzed",
               "analyzer":"diacritical"
            }
         }
     }    
}

Finally, a sample insert is

{
    field: "PEI.H.02354.01."
}

Does anyone have any idea why this is happening and how to solve this?

Answer 1

See the query_string documentation:

Wildcarded terms are not analyzed by default — they are lowercased (lowercase_expanded_terms defaults to true) but no further analysis is done

your stored data is broken up into two terms:

curl -XGET 'localhost:9200/myindex/_analyze?analyzer=diacritical&pretty' -d 'PEI.H.02354.01'
{
  "tokens" : [ {
    "token" : "pei.h",
    "start_offset" : 0,
    "end_offset" : 5,
    "type" : "<ALPHANUM>",
    "position" : 1
  }, {
    "token" : "02354.01",
    "start_offset" : 6,
    "end_offset" : 14,
    "type" : "<NUM>",
    "position" : 2
  } ]
}

but as your search term with a wildcard is only turned into pei.h.02354.01.* it won't match.

however with analyze_wildcard set to true, you do get hits:

curl -XGET "http://localhost:9200/myindex/testType/_search?pretty" -d'
> {
>    "query":{
>       "query_string":{
>          "query":"field:PEI.H.02354.01.*",
>          "default_operator":"AND",
>          "analyze_wildcard": true
>       }
>    }
> }'
{
  "took" : 5,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 2,
    "max_score" : 1.4142135,