Menu
I'm trying to get ElasticSearch to index content for my autocomplete service, using Completion Suggesters in v1.4x. I was following advice from ElasticSearch - You Complete Me and am using the Go Client olivere/elastic.
My index method looks a bit like this:
func IndexVehicle(client *elastic.Client, vehicle Vehicle) (bool, error) {
// See if it exists already
fetch, err := client.Get().
Index(vehicleIndex).
Type("vehicle").
Id(vehicle.Id).
Do()
if err != nil || fetch.Found {
return false, err
}
vehicleName := fmt.Sprintf("%s %s (%s) %s", vehicle.Make, vehicle.Model, vehicle.Model_year, vehicle.Primary_fuel)
suggest := elastic.NewSuggestField()
suggest.Input(vehicle.Make, vehicle.Model, vehicle.Primary_fuel, vehicle.Model_year).
Output(vehicleName).
Payload(vehicle)
// Go forth and save
put, err := client.Index().
Index(vehicleIndex).
Type("vehicle").
Id(vehicle.Id).
Debug(true).Pretty(true).
BodyJson(indexBody{Name: vehicleName, Suggest: suggest}).
Do()
if err != nil {
return false, err
}
return put.Created, nil
}
Now what is weird, is that in some of my tests this method works fine and items will be indexed. In other tests after wiping and rebuilding the index the tests will fail:
Earlier tests look like this in the debug:
2014/12/15 14:11:37 PUT /vehicle/vehicle/369f96459b340507c4688740da3bfe1a?pretty=true HTTP/1.1
Host: localhost:9200
User-Agent: elastic/1.3.1 (darwin-amd64)
Transfer-Encoding: chunked
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip
{"name":"American Motors Corporation Eagle 4WD (1986) regular","suggest":{"input":["American Motors Corporation","Eagle 4WD","regular","1986"],"output":"American Motors Corporation Eagle 4WD (1986) regular","payload":{"make":"American Motors Corporation","model_year":"1986","model":"Eagle 4WD","primary_fuel":"regular","vehicle_class":"Special Purpose Vehicle 4WD","transmission":"Automatic 3-spd","displacement":"4.2","drive":"4-Wheel or All-Wheel Drive","city_mpg":"15.0","highway_mpg":"19.0","comb_mpg":"17.0"}}}
2014/12/15 14:11:37 HTTP/1.1 201 Created
Content-Length: 134
Content-Type: application/json; charset=UTF-8
{
"_index" : "vehicle",
"_type" : "vehicle",
"_id" : "369f96459b340507c4688740da3bfe1a",
"_version" : 1,
"created" : true
}
But in later tests, doing this same thing leads to errors. The err returned from IndexVehicle() in later tests is:
E
Errors:
* /Users/phil/go/src/github.com/ride/autocomplete/vehicle_test.go
Line 79: - elastic: Error 400: ElasticsearchIllegalArgumentException[No feature for name [vehicle]]
goroutine 245 [running]:
github.com/ride/autocomplete.func·033()
/Users/phil/go/src/github.com/ride/autocomplete/vehicle_test.go:79 +0x249
github.com/ride/autocomplete.useIndex(0x499e98)
/Users/phil/go/src/github.com/ride/autocomplete/test_helper.go:18 +0x55
github.com/ride/autocomplete.func·034()
/Users/phil/go/src/github.com/ride/autocomplete/vehicle_test.go:96 +0x2a
github.com/jtolds/gls._m(0x0, 0xc2080ae9e0)
/Users/phil/go/src/github.com/jtolds/gls/stack_tags.go:70 +0x32
github.com/jtolds/gls.markS(0x0, 0xc2080ae9e0)
/Users/phil/go/src/github.com/jtolds/gls/stack_tags.go:21 +0x32
github.com/jtolds/gls.addStackTag(0x0, 0xc2080ae9e0)
/Users/phil/go/src/github.com/jtolds/gls/stack_tags.go:18 +0x3e
github.com/jtolds/gls.(*ContextManager).SetValues(0xc20801e080, 0xc2080b31d0, 0xc2080ae9e0)
/Users/phil/go/src/github.com/jtolds/gls/context.go:98 +0x503
github.com/ride/autocomplete.TestSearchForVehicles(0xc20806a480)
/Users/phil/go/src/github.com/ride/autocomplete/vehicle_test.go:97 +0x243
testing.tRunner(0xc20806a480, 0x5be890)
/opt/boxen/homebrew/Cellar/go/1.4/libexec/src/testing/testing.go:447 +0xbf
created by testing.RunTests
/opt/boxen/homebrew/Cellar/go/1.4/libexec/src/testing/testing.go:555 +0xa8b
goroutine 1 [chan receive]:
testing.RunTests(0x49a078, 0x5be800, 0x7, 0x7, 0x67c001)
/opt/boxen/homebrew/Cellar/go/1.4/libexec/src/testing/testing.go:556 +0xad6
testing.(*M).Run(0xc2080463c0, 0x5c9b20)
/opt/boxen/homebrew/Cellar/go/1.4/libexec/src/testing/testing.go:485 +0x6c
main.main()
github.com/ride/autocomplete/_test/_testmain.go:64 +0x1d5
goroutine 208 [chan receive]:
github.com/olivere/elastic.(*Client).pinger(0xc208106d40)
/Users/phil/go/src/github.com/olivere/elastic/client.go:79 +0x6b
created by github.com/olivere/elastic.NewClient
/Users/phil/go/src/github.com/olivere/elastic/client.go:60 +0x26e
goroutine 248 [runnable]:
net/http.(*persistConn).readLoop(0xc20802e4d0)
/opt/boxen/homebrew/Cellar/go/1.4/libexec/src/net/http/transport.go:928 +0x9ce
created by net/http.(*Transport).dialConn
/opt/boxen/homebrew/Cellar/go/1.4/libexec/src/net/http/transport.go:660 +0xc9f
goroutine 98 [chan receive]:
github.com/olivere/elastic.(*Client).pinger(0xc208033e00)
/Users/phil/go/src/github.com/olivere/elastic/client.go:79 +0x6b
created by github.com/olivere/elastic.NewClient
/Users/phil/go/src/github.com/olivere/elastic/client.go:60 +0x26e
goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/opt/boxen/homebrew/Cellar/go/1.4/libexec/src/runtime/asm_amd64.s:2232 +0x1
goroutine 44 [chan receive]:
github.com/olivere/elastic.(*Client).pinger(0xc2080332c0)
/Users/phil/go/src/github.com/olivere/elastic/client.go:79 +0x6b
created by github.com/olivere/elastic.NewClient
/Users/phil/go/src/github.com/olivere/elastic/client.go:60 +0x26e
goroutine 249 [select]:
net/http.(*persistConn).writeLoop(0xc20802e4d0)
/opt/boxen/homebrew/Cellar/go/1.4/libexec/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
/opt/boxen/homebrew/Cellar/go/1.4/libexec/src/net/http/transport.go:661 +0xcbc
goroutine 54 [chan receive]:
github.com/olivere/elastic.(*Client).pinger(0xc208032f80)
/Users/phil/go/src/github.com/olivere/elastic/client.go:79 +0x6b
created by github.com/olivere/elastic.NewClient
/Users/phil/go/src/github.com/olivere/elastic/client.go:60 +0x26e
goroutine 76 [chan receive]:
github.com/olivere/elastic.(*Client).pinger(0xc208032e00)
/Users/phil/go/src/github.com/olivere/elastic/client.go:79 +0x6b
created by github.com/olivere/elastic.NewClient
/Users/phil/go/src/github.com/olivere/elastic/client.go:60 +0x26e
goroutine 250 [chan receive]:
github.com/olivere/elastic.(*Client).pinger(0xc208106c40)
/Users/phil/go/src/github.com/olivere/elastic/client.go:79 +0x6b
created by github.com/olivere/elastic.NewClient
/Users/phil/go/src/github.com/olivere/elastic/client.go:60 +0x26e
goroutine 120 [chan receive]:
github.com/olivere/elastic.(*Client).pinger(0xc208106b00)
/Users/phil/go/src/github.com/olivere/elastic/client.go:79 +0x6b
created by github.com/olivere/elastic.NewClient
/Users/phil/go/src/github.com/olivere/elastic/client.go:60 +0x26e
goroutine 142 [chan receive]:
github.com/olivere/elastic.(*Client).pinger(0xc208106e00)
/Users/phil/go/src/github.com/olivere/elastic/client.go:79 +0x6b
created by github.com/olivere/elastic.NewClient
/Users/phil/go/src/github.com/olivere/elastic/client.go:60 +0x26e
goroutine 164 [chan receive]:
github.com/olivere/elastic.(*Client).pinger(0xc208106b80)
/Users/phil/go/src/github.com/olivere/elastic/client.go:79 +0x6b
created by github.com/olivere/elastic.NewClient
/Users/phil/go/src/github.com/olivere/elastic/client.go:60 +0x26e
goroutine 186 [chan receive]:
github.com/olivere/elastic.(*Client).pinger(0xc208106d00)
/Users/phil/go/src/github.com/olivere/elastic/client.go:79 +0x6b
created by github.com/olivere/elastic.NewClient
/Users/phil/go/src/github.com/olivere/elastic/client.go:60 +0x26e
goroutine 230 [chan receive]:
github.com/olivere/elastic.(*Client).pinger(0xc208106dc0)
/Users/phil/go/src/github.com/olivere/elastic/client.go:79 +0x6b
created by github.com/olivere/elastic.NewClient
/Users/phil/go/src/github.com/olivere/elastic/client.go:60 +0x26e
Possibly the more important part of that backtrace is this:
Error 400: ElasticsearchIllegalArgumentException[No feature for name [vehicle]]
So, not sure what the heck is going wrong here. I have a 1 second timeout on the index, because this client doesn't support "wait for green" logic [yet].
func ResetVehicleIndex(client *elastic.Client) (err error) {
if _, err = client.DeleteIndex(vehicleIndex).Do(); err != nil {
return
}
if _, err = EnsureVehicleIndex(client); err != nil {
return
}
// TODO: This is awful. Switch to "wait for green" when elastic client supports it
time.Sleep(time.Second * 1)
return nil
}
This has worked for most other tests, as elasticsearch seems to get ready in a second or so, but these tests are consistently erroring regardless of the wait time in that bit of code.
Any ideas? I might need to edit the question to add more code or better explanations, but I'll reply super quickly to any questions especially if you ping me on Twitter @philsturgeon. I'm really stuck on this.
495
In Elasticsearch, an index (plural: indices) contains a schema and can have one or more shards and replicas. An Elasticsearch index is divided into shards and each shard is an instance of a Lucene index. Indices are used to store the documents in dedicated data structures corresponding to the data type of fields.
The Go client constructor accepts the elasticsearch.Config {} type, which provides a number of options to control the behaviour: $ go doc -short github.com/elastic/go-elasticsearch/v7.Config type Config struct { Addresses []string // A list of Elasticsearch nodes to use. // ... } Let's review the available options and examples of their usage.
The fact that it's so easy to handle this case in the Python client may be an argument against adding this feature to Elasticsearch itself, if the error can be handled quickly and easily in our official clients.
Use the APIKey field for authentication with the API keys, which are easier to manage via the Elasticsearch API or Kibana than usernames and passwords. When using Elasticsearch Service on Elastic Cloud, you can point the client to the cluster by using the Cloud ID:
That exception is only triggered in one place, which is during a Get Index API call. Which means your vehicle ID must be null here:
fetch, err := client.Get().
Index(vehicleIndex).
Type("vehicle").
Id(vehicle.Id). //<-- this
Do()
You are trying to do a Get Document API, which follows the format of GET /{index}/{type}/{id}. However, your client doesn't make a distinction between Get Document and Get Index API calls...and it doesn't validate that your parameters are non-null.
So if a null vehicle.Id is passed to the Get method, your final URL will actually be GET /{index}/{type}/
From Elasticsearch's point of view, this is no longer a Get Document API call...it's actually a Get Index call, which has the following format: GET /{index}/{feature}. Feature can be one of: _settings, _mappings, _aliases or _warmers.
Because vehicle is not one of those features, ES is throwing an exception and spewing. You can verify this from the console:
curl -XGET localhost:9200/my_index/vehicle/
115
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
