Rails4 production elasticsearch error - failed to find geo_point, but works in Development

Question

i have a fine running Rails 4 app using elastic search and searchkick.I have a geosearch by setting up geo_point and it works great in dev, but after deploying the same code and verifying the indices in production, on Digital Ocean with 3GB RAM and Ubuntu-16 OS, its failing with the error -

A Searchkick::InvalidQueryError occurred in home#show_by_location_and_event:

      [400] {"error":{"root_cause":[{"type":"query_parsing_exception","reason":"failed to find geo_point field [location]","index":"halls_production_20180520121711247","line":1,"col":724}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"halls_production_20180520121711247","node":"Mn7tHSKLQWSB1N8Sypb6Qg","reason":{"type":"query_parsing_exception","reason":"failed to find geo_point field [location]","index":"halls_production_20180520121711247","line":1,"col":724}}]},"status":400}
      app/controllers/home_controller.rb:168:in `show_by_location_and_event'

home_controller.rb

  lat_lng = Geocoder.coordinates(@location)
  @event_name = params[:event_name]
  ###XXXX FAILING HERE AT BELOW LINE
  @halls =  Hall.get_completed_halls_only.search @location ,
              where: { 
                location: {  near: lat_lng,within: "20km"} ,
                workflow_state: "accepted",
                active: "true"                        
              }

models/hall.rb

##added geo_point
##hall has one address association
searchkick word_start: [:name, :slug, :description, :facebook_link],
             word_middle: [:name, :slug, :description, :facebook_link],
             text_start: [:name, :slug] , locations: ["location"]

  def search_data
    {
      location: [self.address.latitude, self.address.longitude],
    }
  end

SEARCH query in rails console for both development/production

    ###=====================query works in development rails console=================

    2.2.4 :023 > @halls =  Hall.get_completed_halls_only.search "Mumbai" , where: { location: {  near: [72.8776559,19.0759837],within: "20km"}}
      Hall Search (10.0ms)  curl http://localhost:9200/halls_development/_search?pretty -d '{"query":{"bool":{"must":{"dis_max":{"queries":[{"match":{"_all":{"query":"Mumbai","boost":10,"operator":"and","analyzer":"searchkick_search"}}},{"match":{"_all":{"query":"Mumbai","boost":10,"operator":"and","analyzer":"searchkick_search2"}}},{"match":{"_all":{"query":"Mumbai","boost":1,"operator":"and","analyzer":"searchkick_search","fuzziness":1,"prefix_length":0,"max_expansions":3,"fuzzy_transpositions":true}}},{"match":{"_all":{"query":"Mumbai","boost":1,"operator":"and","analyzer":"searchkick_search2","fuzziness":1,"prefix_length":0,"max_expansions":3,"fuzzy_transpositions":true}}}]}},"filter":[{"geo_distance":{"location":[19.0759837,72.8776559],"distance":"20km"}}]}},"size":1000,"from":0,"timeout":"11s","_source":false}'

     => #<Searchkick::Results:0x00000005bed2a0 @klass=Hall(id: integer, name: string, homepage_link: string, description: text, hall_request_id: integer, active: boolean, announcements_count: integer, comments_count: integer, managers_count: integer, pictures_count: integer, videos_count: integer, testimonials_count: integer, created_at: datetime, updated_at: datetime, capacity_in_persons: string, recomendations_count: integer, checkins_count: integer, deleted_at: datetime, additional_details: text, slug: string, food_type: string, facebook_link: string, workflow_state: string, impressions_count: integer, hall_promotions_count: integer, venue_packages_count: integer), @response={"took"=>5, "timed_out"=>false, "_shards"=>{"total"=>5, "successful"=>5, "failed"=>0}, "hits"=>{"total"=>0, "max_score"=>nil, "hits"=>[]}}, @options={:page=>1, :per_page=>1000, :padding=>0, :load=>true, :includes=>nil, :json=>false, :match_suffix=>"analyzed", :highlighted_fields=>[], :misspellings=>true}> 



 ##====================same above query do not work in production rails console - as a part of debugging that i did==========
     2.2.4 :066 > @halls =  Hall.get_completed_halls_only.search "Mumbai" , where: { location: {  near: [72.8776559,19.0759837],within: "20km"}}                                       
      Hall Search (297.8ms)  curl http://localhost:9200/halls_production/_search?pretty -d '{"query":{"bool":{"must":{"dis_max":{"queries":[{"match":{"_all":{"query":"Mumbai","boost":10,"operator":"and","analyzer":"searchkick_search"}}},{"match":{"_all":{"query":"Mumbai","boost":10,"operator":"and","analyzer":"searchkick_search2"}}},{"match":{"_all":{"query":"Mumbai","boost":1,"operator":"and","analyzer":"searchkick_search","fuzziness":1,"prefix_length":0,"max_expansions":3,"fuzzy_transpositions":true}}},{"match":{"_all":{"query":"Mumbai","boost":1,"operator":"and","analyzer":"searchkick_search2","fuzziness":1,"prefix_length":0,"max_expansions":3,"fuzzy_transpositions":true}}}]}},"filter":[{"geo_distance":{"location":[19.0759837,72.8776559],"distance":"20km"}}]}},"size":1000,"from":0,"timeout":"11s","_source":false}'

    Searchkick::InvalidQueryError: [400] {"error":{"root_cause":[{"type":"query_parsing_exception","reason":"failed to find geo_point field [location]","index":"halls_production_20180520121711247","line":1,"col":677}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"halls_production_20180520121711247","node":"Mn7tHSKLQWSB1N8Sypb6Qg","reason":{"type":"query_parsing_exception","reason":"failed to find geo_point field [location]","index":"halls_production_20180520121711247","line":1,"col":677}}]},"status":400}
            from /home/deployer/apps/halls/shared/bundle/ruby/2.2.0/gems/searchkick-2.2.0/lib/searchkick/query.rb:184:in `handle_error'
            from /home/deployer/apps/halls/shared/bundle/ruby/2.2.0/gems/searchkick-2.2.0/lib/searchkick/query.rb:85:in `rescue in execute'
            from /home/deployer/apps/halls/shared/bundle/ruby/2.2.0/gems/searchkick-2.2.0/lib/searchkick/query.rb:78:in `execute'
            from /home/deployer/apps/halls/shared/bundle/ruby/2.2.0/gems/searchkick-2.2.0/lib/searchkick.rb:99:in `search'
            from /home/deployer/apps/halls/shared/bundle/ruby/2.2.0/gems/searchkick-2.2.0/lib/searchkick/model.rb:29:in `searchkick_search'
            from /home/deployer/apps/halls/shared/bundle/ruby/2.2.0/gems/activerecord-4.2.6/lib/active_record/relation/delegation.rb:70:in `block in search'
            from /home/deployer/apps/halls/shared/bundle/ruby/2.2.0/gems/activerecord-4.2.6/lib/active_record/relation.rb:302:in `scoping'
            from /home/deployer/apps/halls/shared/bundle/ruby/2.2.0/gems/activerecord-4.2.6/lib/active_record/relation/delegation.rb:70:in `search'


###=====response for elasticsearch on the halls_production index
deployer@Hallpick:~/apps/hallpick/current$ curl 'localhost:9200/halls_production'
{"halls_production_20180527160622250":{"aliases":{"halls_production":{}},"mappings":{"hall":{"_all":{"analyzer":"searchkick_index"},"dynamic_templates":[{"string_template":{"mapping":{"include_in_all":true,"ignore_above":30000,"index":"not_analyzed","type":"string","fields":{"analyzed":{"index":"analyzed","type":"string"}}},"match":"*","match_mapping_type":"string"}}],"properties":{"location":{"type":"geo_point"},"locations":{"type":"geo_point"},"name":{"type":"string","index":"not_analyzed","fields":{"analyzed":{"type":"string","analyzer":"searchkick_index"},"text_start":{"type":"string","analyzer":"searchkick_text_start_index"},"word_middle":{"type":"string","analyzer":"searchkick_word_middle_index"},"word_start":{"type":"string","analyzer":"searchkick_word_start_index"}},"ignore_above":30000},"slug":{"type":"string","index":"not_analyzed","fields":{"analyzed":{"type":"string","analyzer":"searchkick_index"},"text_start":{"type":"string","analyzer":"searchkick_text_start_index"},"word_middle":{"type":"string","analyzer":"searchkick_word_middle_index"},"word_start":{"type":"string","analyzer":"searchkick_word_start_index"}},"ignore_above":30000}}},"_default_":{"_all":{"analyzer":"searchkick_index"},"dynamic_templates":[{"string_template":{"mapping":{"include_in_all":true,"ignore_above":30000,"index":"not_analyzed","type":"string","fields":{"analyzed":{"index":"analyzed","type":"string"}}},"match":"*","match_mapping_type":"string"}}],"properties":{"location":{"type":"geo_point"},"name":{"type":"string","index":"not_analyzed","fields":{"analyzed":{"type":"string","analyzer":"searchkick_index"},"text_start":{"type":"string","analyzer":"searchkick_text_start_index"},"word_middle":{"type":"string","analyzer":"searchkick_word_middle_index"},"word_start":{"type":"string","analyzer":"searchkick_word_start_index"}},"ignore_above":30000},"slug":{"type":"string","index":"not_analyzed","fields":{"analyzed":{"type":"string","analyzer":"searchkick_index"},"text_start":{"type":"string","analyzer":"searchkick_text_start_index"},"word_middle":{"type":"string","analyzer":"searchkick_word_middle_index"},"word_start":{"type":"string","analyzer":"searchkick_word_start_index"}},"ignore_above":30000}}}},"settings":{"index":{"creation_date":"1527437182297","analysis":{"filter":{"searchkick_suggest_shingle":{"max_shingle_size":"5","type":"shingle"},"searchkick_edge_ngram":{"type":"edgeNGram","min_gram":"1","max_gram":"50"},"searchkick_index_shingle":{"token_separator":"","type":"shingle"},"searchkick_search_shingle":{"token_separator":"","output_unigrams_if_no_shingles":"true","output_unigrams":"false","type":"shingle"},"searchkick_stemm r":{"type":"snowball","language":"English"},"searchkick_ngram":{"type":"nGram","min_gram":"1","max_gram":"50"}},"analyzer":{"searchkick_word_start_index":{"filter":["lowercase","asciifolding","searchkick_edge_ngram"],"type":"custom","tokenizer":"standard"},"searchkick_keyword":{"filter":["lowercase","searchkick_stemmer"],"type":"custom","tokenizer":"keyword"},"searchkick_text_end_index":{"filter":["lowercase","asciifolding","reverse","searchkick_edge_ngram","reverse"],"type":"custom","tokenizer":"keyword"},"searchkick_search2":{"filter":["standard","lowercase","asciifolding","searchkick_stemmer"],"char_filter":["ampersand"],"type":"custom","tokenizer":"standard"},"searchkick_word_middle_index":{"filter":["lowercase","asciifolding","searchkick_ngram"],"type":"custom","tokenizer":"standard"},"searchkick_search":{"filter":["standard","lowercase","asciifolding","searchkick_search_shingle","searchkick_stemmer"],"char_filter":["ampersand"],"type":"custom","tokenizer":"standard"},"searchkick_text_start_index":{"filter":["lowercase","asciifolding","searchkick_edge_ngram"],"type":"custom","tokenizer":"keyword"},"searchkick_word_end_index":{"filter":["lowercase","asciifolding","reverse","searchkick_edge_ngram","reverse"],"type":"custom","tokenizer":"standard"},"searchkick_word_search":{"filter":["lowercase","asciifolding"],"type":"custom","tokenizer":"standard"},"searchkick_autocomplete_search":{"filter":["lowercase","asciifolding"],"type":"custom","tokenizer":"keyword"},"searchkick_suggest_index":{"filter":["lowercase","asciifolding","searchkick_suggest_shingle"],"type":"custom","tokenizer":"standard"},"searchkick_text_middle_index":{"filter":["lowercase","asciifolding","searchkick_ngram"],"type":"custom","tokenizer":"keyword"},"searchkick_index":{"filter":["standard","lowercase","asciifolding","searchkick_index_shingle","searchkick_stemmer"],"char_filter":["ampersand"],"type":"custom","tokenizer":"standard"}},"char_filter":{"ampersand":{"type":"mapping","mappings":["&=> and "]}}},"number_of_shards":"5","number_of_replicas":"1","uuid":"gcfBQAZuTpW8YqMVdw2ALg","version":{"created":"2040699"}}},"warmers":{}}}deployer@Hallpick:~/apps/hallpick/current$ 

is it because my index is empty as there is no data in the production.? I think i have provided all the required details that i can.Still wondering what is the issue.

Answer 1

i solved the above error and let me share how.

i think there was some reason because of the versions of Elasticsearch on my dev and prod.

========on dev==========
Loading development environment (Rails 4.2.6)
2.2.4 :001 > Elasticsearch::Model.client.info
 => {"name"=>"Sunpyre", "cluster_name"=>"elasticsearch", "version"=>{"number"=>"2.3.5", "build_hash"=>"90f433701e64ccd01edbb4", "build_timestamp"=>"2016-07-27T10:36:52Z", "build_snapshot"=>false, "lucene_version"=>"5.5.0"}, "tagline"=>"You Know, for Search"} 
2.2.4 :002 > 

=======on prod=========
Loading production environment (Rails 4.2.6)
2.2.0 :001 > Elasticsearch::Model.client.info
 => {"name"=>"Production Node", "cluster_name"=>"elasticsearch", "cluster_uuid"=>"NCPUn7V084cpIA", "version"=>{"number"=>"2.4.6", "build_hash"=>"5376dca9f70f3abe0ace8240fd", "build_timestamp"=>"2017-07-18T12:17:44Z", "build_snapshot"=>false, "lucene_version"=>"5.5.4"}, "tagline"=>"You Know, for Search"} 
2.2.0 :002 > 

So the mappings like the below worked in dev but not on prod.

====working on dev but not on prod=====

searchkick word_start: [:name, :slug],
             word_middle: [:name, :slug],
             text_start: [:name, :slug],
             locations: ["location"]

But when i added explicit mappings, it worked on prod..also on dev(needless to say)

searchkick merge_mappings: true, 
  mappings: {hall: 
      {properties: 

        {
          locations: 
          {
            type:"geo_point"

          }

        }

      }

  },
  locations: ["location"],
  word_start: [:name, :slug],
  word_middle: [:name, :slug],
  text_start: [:name, :slug]

  mapping do
    indexes :location, type: 'geo_point'
  end

I think all the custom mappings was the thing needed on the 2.4 version which is not visible on ChangeLog.

Hope this helps someone.