Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Elasticsearch 7 The [standard] token filter has been removed

I am attempting to upgrade to Elasticsearch v7 (I'm using the ruby/rails client), and upon doing so and fixing several stuff, I run across the following error

Elasticsearch::Transport::Transport::Errors::BadRequest:
  [400] {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The [standard] token filter has been removed."}],"type":"illegal_argument_exception","reason":"The [standard] token filter has been removed."},"status":400}

Upon checking the breaking changes, it is indeed mentioned that

The standard token filter has been removed because it doesn’t change anything in the stream.

I am not sure how I should reflect this in my config. If I understand correctly, this error may come from my custom phrase suggester

{
  "analysis": {
    "filter": {
      "shingle": {
        "type": "shingle",
        "min_shingle_size": 2,
        "max_shingle_size": 3
      }
    },
    "analyzer": {
      "trigram": {
        "type": "custom",
        "tokenizer": "standard",
        "filter": ["standard", "shingle"]
      },
      "reverse": {
        "type": "custom",
        "tokenizer": "standard",
        "filter": ["standard", "reverse"]
      }
    }
  }
}

Should I just remove the tokenizer field ? Maybe they forgot to update it but this is still what's mentionned in the [Elasticsearch documentation of the suggester][1]

If the problem is not coming from there, where should I look into ?

  • ES 7.3.2
  • elasticsearch-api-7.3.0 | elasticsearch-transport 7.3.0
  • elasticsearch-model 7.0.0 | elasticsearch-rails 7.0.0
like image 623
Cyril Duchon-Doris Avatar asked Oct 02 '19 12:10

Cyril Duchon-Doris


1 Answers

Credit goes to @Rob in the comments:

{
  "analysis": {
    "filter": {
      "shingle": {
        "type": "shingle",
        "min_shingle_size": 2,
        "max_shingle_size": 3
      }
    },
    "analyzer": {
      "trigram": {
        "type": "custom",
        "tokenizer": "standard",
        "filter": ["shingle"]
      },
      "reverse": {
        "type": "custom",
        "tokenizer": "standard",
        "filter": ["reverse"]
      }
    }
  }
}
like image 200
Cyril Duchon-Doris Avatar answered Nov 09 '22 07:11

Cyril Duchon-Doris