Searchkick index is empty after reindexing from model

Question

When I try to reindex records from inside a model it ends up setting an empty index on elasticsearch server.

So, I have a Course model which has many Instructors. Now, I have this callback method inside instructor.rb model which tries to reindex whenever something is updated on Instructor.

class Instructor < ActiveRecord::Base
belongs_to :teacher, :class_name => "User", :foreign_key => "user_id"
belongs_to :instructorable, polymorphic: true
after_commit :reindex_course

private

  def reindex_course
     Course.reindex_course
  end
end

course.rb model looks like this:

    class Course < ActiveRecord::Base
        has_many :instructors, as: :instructorable, dependent: :destroy

searchkick word_start: [ :name, :description, :instructor_name]

def search_data
      {
      name: name,
      description: description,
      searchable: searchable,
      }.merge(
        instructor_name: instructors.includes(:teacher).pluck(:'users.name').reject(&:nil?),
      institute_id: creator.try(:institute_id)
)
            def self.reindex_course
                index = Course.reindex(async: true, refresh_interval: "5s")
                Course.search_index.promote(index[:index_name], update_refresh_interval: true)
                Course.search_index.clean_indices
              end
    end

Creating a new Instructor leads to reindexing of records

Instructor.create(user_id: 4009,instructorable_id: 4792, instructorable_type:"ScheduledCourse")
    ETHON: performed EASY effective_url=http://xx:[email protected]/elasticsearch/_aliases response_code=200 return_code=ok total_time=0.630504
    ETHON: performed EASY effective_url=http://xx:[email protected]/elasticsearch/courses_development_20171204162604885 response_code=200 return_code=ok total_time=0.922072
    ETHON: performed EASY effective_url=http://xx:[email protected]/elasticsearch/_alias/courses_development response_code=200 return_code=ok total_time=0.29790799999999995
       (5.0ms)  SELECT MIN("courses"."id") FROM "courses" LEFT OUTER JOIN "users" ON "users"."id" = "courses"."creator_id" LEFT OUTER JOIN "units" ON "units"."course_id" = "courses"."id" LEFT OUTER JOIN "instructors" ON "instructors"."instructorable_id" = "courses"."id" AND "instructors"."instructorable_type" = $1  [["instructorable_type", "Course"]]
       (4.4ms)  SELECT MAX("courses"."id") FROM "courses" LEFT OUTER JOIN "users" ON "users"."id" = "courses"."creator_id" LEFT OUTER JOIN "units" ON "units"."course_id" = "courses"."id" LEFT OUTER JOIN "instructors" ON "instructors"."instructorable_id" = "courses"."id" AND "instructors"."instructorable_type" = $1  [["instructorable_type", "Course"]]
    Enqueued Searchkick::BulkReindexJob (Job ID: d710eff3-5e39-4347-b02a-5459bfd4b1be) to Sidekiq(searchkick) with arguments: {:class_name=>"Course", :index_name=>"courses_development_20171204162604885", :batch_id=>1, :min_id=>2, :max_id=>1001}
    Enqueued Searchkick::BulkReindexJob (Job ID: f0bf5866-f3da-42dd-9132-2de23735fbf4) to Sidekiq(searchkick) with arguments: {:class_name=>"Course", :index_name=>"courses_development_20171204162604885", :batch_id=>2, :min_id=>1002, :max_id=>2001}
    Enqueued Searchkick::BulkReindexJob (Job ID: 348e07b8-a85a-4a9f-a2f2-d2b70416d2bf) to Sidekiq(searchkick) with arguments: {:class_name=>"Course", :index_name=>"courses_development_20171204162604885", :batch_id=>3, :min_id=>2002, :max_id=>3001}
    Enqueued Searchkick::BulkReindexJob (Job ID: e40f2a33-e2f7-4f77-9416-920d84e285e7) to Sidekiq(searchkick) with arguments: {:class_name=>"Course", :index_name=>"courses_development_20171204162604885", :batch_id=>4, :min_id=>3002, :max_id=>4001}
    Enqueued Searchkick::BulkReindexJob (Job ID: 18b357c1-cd53-4836-8aaf-4c97dde56281) to Sidekiq(searchkick) with arguments: {:class_name=>"Course", :index_name=>"courses_development_20171204162604885", :batch_id=>5, :min_id=>4002, :max_id=>5001}
    ETHON: performed EASY effective_url=http://xx:[email protected]/elasticsearch/courses_development_20171204162604885/_settings response_code=200 return_code=ok total_time=0.372641
    ETHON: performed EASY effective_url=http://xx:[email protected]/elasticsearch/_alias/courses_development response_code=200 return_code=ok total_time=0.333225
    ETHON: performed EASY effective_url=http://xx:[email protected]/elasticsearch/_aliases response_code=200 return_code=ok total_time=0.42405099999999996
    ETHON: performed EASY effective_url=http://xx:[email protected]/elasticsearch/_aliases response_code=200 return_code=ok total_time=0.382619
    ETHON: performed EASY effective_url=http://xx:[email protected]/elasticsearch/courses_development_20171204162504480 response_code=200 return_code=ok total_time=0.408705

Now when I run this query:

Searchkick.search search_params[:search], operator: "or",
           index_name: [Course],
           where:{institute_id: institute_id, searchable: true},
           fields: ['name^100','instructor_name^100','description^50'],
           match: :word_start, misspellings: {edit_distance: 2},
           page: search_params[:page],per_page: 20,
           order: {_score: :desc}

I get an empty response:

ETHON: performed EASY effective_url=http://xx:[email protected]/elasticsearch/courses_development/_search response_code=200 return_code=ok total_time=0.733799
  Search (738.0ms)  curl http://xx.xxx.xx.xxx:80/courses_development/_search?pretty -d '{"query":{"bool":{"must":{"dis_max":{"queries":[{"bool":{"must":{"bool":{"should":[{"match":{"name.word_start":{"query":"Elasticsearch","boost":1000.0,"operator":"or","analyzer":"searchkick_word_search"}}},{"match":{"name.word_start":{"query":"Elasticsearch","boost":100.0,"operator":"or","analyzer":"searchkick_word_search","fuzziness":2,"prefix_length":0,"max_expansions":3,"fuzzy_transpositions":true}}}]}},"should":{"match":{"name.analyzed":{"query":"Elasticsearch","boost":1000.0,"operator":"or","analyzer":"searchkick_word_search"}}}}},{"bool":{"must":{"bool":{"should":[{"match":{"instructor_name.word_start":{"query":"Elasticsearch","boost":1000.0,"operator":"or","analyzer":"searchkick_word_search"}}},{"match":{"instructor_name.word_start":{"query":"Elasticsearch","boost":100.0,"operator":"or","analyzer":"searchkick_word_search","fuzziness":2,"prefix_length":0,"max_expansions":3,"fuzzy_transpositions":true}}}]}},"should":{"match":{"instructor_name.analyzed":{"query":"Elasticsearch","boost":1000.0,"operator":"or","analyzer":"searchkick_word_search"}}}}},{"bool":{"must":{"bool":{"should":[{"match":{"description.word_start":{"query":"Elasticsearch","boost":500.0,"operator":"or","analyzer":"searchkick_word_search"}}},{"match":{"description.word_start":{"query":"Elasticsearch","boost":50.0,"operator":"or","analyzer":"searchkick_word_search","fuzziness":2,"prefix_length":0,"max_expansions":3,"fuzzy_transpositions":true}}}]}},"should":{"match":{"description.analyzed":{"query":"Elasticsearch","boost":500.0,"operator":"or","analyzer":"searchkick_word_search"}}}}}]}},"filter":[{"term":{"institute_id":113}},{"term":{"searchable":true}}]}},"size":1000,"from":0,"timeout":"11s","_source":false}'
=> #<Searchkick::Results:0x007fcb8c9cb378
 @klass=nil,
 @options=
  {:page=>1,
   :per_page=>1000,
   :padding=>0,
   :load=>true,
   :includes=>nil,
   :model_includes=>nil,
   :json=>false,
   :match_suffix=>:word_start,
   :highlighted_fields=>[],
   :misspellings=>true},
 @response=
  {"took"=>1,
   "timed_out"=>false,
   "_shards"=>{"total"=>5, "successful"=>5, "skipped"=>0, "failed"=>0},
   "hits"=>{"total"=>0, "max_score"=>nil, "hits"=>[]}}>

However, when i run Course.reindex(from console or model) or RAILS_ENV=development bundle exec rake searchkick:reindex CLASS=Course --trace form terminal it again starts working.

[33] pry(main)> Searchkick.search "Elasticsearch", operator: "or",index_name: [Course],where:{institute_id: 113, searchable: true},fields: ['name^100','instructor_name^100','description^50'],match: :word_start,misspellings: {edit_distance: 2}
ETHON: performed EASY effective_url=http://xx:[email protected]/elasticsearch/courses_development/_search response_code=200 return_code=ok total_time=0.57072
  Search (573.5ms)  curl http://xxx.xx.xx.xxx:80/courses_development/_search?pretty -d '{"query":{"bool":{"must":{"dis_max":{"queries":[{"bool":{"must":{"bool":{"should":[{"match":{"name.word_start":{"query":"Elasticsearch","boost":1000.0,"operator":"or","analyzer":"searchkick_word_search"}}},{"match":{"name.word_start":{"query":"Elasticsearch","boost":100.0,"operator":"or","analyzer":"searchkick_word_search","fuzziness":2,"prefix_length":0,"max_expansions":3,"fuzzy_transpositions":true}}}]}},"should":{"match":{"name.analyzed":{"query":"Elasticsearch","boost":1000.0,"operator":"or","analyzer":"searchkick_word_search"}}}}},{"bool":{"must":{"bool":{"should":[{"match":{"instructor_name.word_start":{"query":"Elasticsearch","boost":1000.0,"operator":"or","analyzer":"searchkick_word_search"}}},{"match":{"instructor_name.word_start":{"query":"Elasticsearch","boost":100.0,"operator":"or","analyzer":"searchkick_word_search","fuzziness":2,"prefix_length":0,"max_expansions":3,"fuzzy_transpositions":true}}}]}},"should":{"match":{"instructor_name.analyzed":{"query":"Elasticsearch","boost":1000.0,"operator":"or","analyzer":"searchkick_word_search"}}}}},{"bool":{"must":{"bool":{"should":[{"match":{"description.word_start":{"query":"Elasticsearch","boost":500.0,"operator":"or","analyzer":"searchkick_word_search"}}},{"match":{"description.word_start":{"query":"Elasticsearch","boost":50.0,"operator":"or","analyzer":"searchkick_word_search","fuzziness":2,"prefix_length":0,"max_expansions":3,"fuzzy_transpositions":true}}}]}},"should":{"match":{"description.analyzed":{"query":"Elasticsearch","boost":500.0,"operator":"or","analyzer":"searchkick_word_search"}}}}}]}},"filter":[{"term":{"institute_id":113}}]}},"size":1000,"from":0,"timeout":"11s","_source":false}'
=> #<Searchkick::Results:0x007fcb8f2893a0
 @klass=nil,
 @options=
  {:page=>1,
   :per_page=>1000,
   :padding=>0,
   :load=>true,
   :includes=>nil,
   :model_includes=>nil,
   :json=>false,
   :match_suffix=>:word_start,
   :highlighted_fields=>[],
   :misspellings=>true},
 @response=
  {"took"=>32,
   "timed_out"=>false,
   "_shards"=>{"total"=>5, "successful"=>5, "skipped"=>0, "failed"=>0},
   "hits"=>
    {"total"=>2,
     "max_score"=>19030.512,
     "hits"=>
      [{"_index"=>"courses_development_20171204170257200",
        "_type"=>"course",
        "_id"=>"4792",
        "_score"=>19030.512},
       {"_index"=>"courses_development_20171204170257200",
        "_type"=>"course",
        "_id"=>"4782",
        "_score"=>14400.825}]}}>

Answer 1

I found a quick fix for this issue. calling touch on parent(Course) class of Instructor and other associated model seems to be updating the index correctly.

instructor.rb model looks like this:

:
:
:
after_commit :reindex_course
:
:
def reindex_course
    if self.try(:instructorable).try(:persisted?)
      if self.instructorable_type == "Course"
        self.instructorable.touch
      elsif self.instructorable_type == "ScheduledCourse"
        self.instructorable.course.touch
      end
    end
  end

As the hook works correctly when something happens on Course model, touch seems to be solving the purpose for now.

Answer to the problem remains a mystery to me.

Answer 2

        def self.reindex_course
            index = Course.reindex(async: true, refresh_interval: "5s")
            Course.search_index.promote(index[:index_name], update_refresh_interval: true)
            Course.search_index.clean_indices
          end

Looks like calling Course.search_index.clean_indices in the last line would clean old indices. I think you may want to call it in the first line, or do not call it at all.