Elasticsearch out of sync when overwhelmed on HTTP at test suite

Question

I have a Rails app with an Rspec test suite which has some feature/controller tests depending on ElasticSearch.

When we test the "search" feature around the system (and other features depending on ES) we use a real ES, it works perfectly at development environment when we're running single spec files.

When the suite runs at our CI server it gets weird, because sometimes ES won't keep in sync fast enough for the tests to run successfully.

I have searched for some way to run ES in "syncronous mode", or to wait until ES is ready but haven't found anything so far. I've seen some workarounds using Ruby sleep but it feels unacceptable to me.

How can I guarantee ES synchronicity to run my tests?

How do you deal with ES on your test suite?

Here's one of my tests:

      context "given params page or per_page is set", :elasticsearch do

      let(:params) { {query: "Resultados", page: 1, per_page: 2} }

      before(:each) do
        3.times do |n|
          Factory(:company, account: user.account, name: "Resultados Digitais #{n}")
        end
        sync_companies_index # this is a helper method available to all specs
      end

      it "paginates the results properly" do
        get :index, params
        expect(assigns[:companies].length).to eq 2
      end

    end

Here's my RSpec configure block and ES helper methods:

RSpec.configure do |config|
  config.around :each do |example|
    if example.metadata[:elasticsearch]
      Lead.tire.index.delete # delete the index for a clean environment
      Company.tire.index.delete # delete the index for a clean environment

      example.run
    else
      FakeWeb.register_uri :any, %r(#{Tire::Configuration.url}), body: '{}'
      example.run
      FakeWeb.clean_registry
    end
  end
end

def sync_companies_index
  sync_index_of Company
end

def sync_leads_index
  sync_index_of Lead
end

def sync_index_of(klass)
  mapping = MultiJson.encode(klass.tire.mapping_to_hash, :pretty => Tire::Configuration.pretty)
  klass.tire.index.create(:mappings => klass.tire.mapping_to_hash, :settings => klass.tire.settings)
  "#{klass}::#{klass}Index".constantize.rebuild_index
  klass.index.refresh
end

Thanks for any help!

Answer 1

Your test is confused - it's testing assignment, pagination, and (implicitly) parameter passing. Break it out:

Parameters

let(:tire) { double('tire', :search => :sentinel) }

it 'passes the correct parameters to Companies.tire.search' do
  expected_params = ... # Some transformation, if any, of params
  Companies.stub(:tire).with(tire)
  get :index, params

  expect(tire).to have_received(:search).with(expected_params)
end

Assignment

We are only concerned that the code is taking one value and assigning it to something else, the value is irrelevant.

it 'assigns the search results to companies' do
  Companies.stub(:tire).with(tire)
  get :index, params

  expect(assigns[:companies]).to eq :sentinel
end

Pagination

This is the tricky bit. You don't own the ES API, so you shouldn't stub it, but you also can't use a live instance of ES because you can't trust it to be reliable in all testing scenarios, it's just an HTTP API after all (this is the fundamental issue you're having). Gary Bernhardt tackled this issue in one of his excellent screencasts - you simply have to fake out the HTTP calls. Using VCR:

VCR.use_cassette :tire_companies_search do
  get :index, params
  search_result_length = assigns[:companies].length

  expect(search_result_length).to eq 2
end

Run this once successfully then forever more use the cassette (which is simply a YAML file of the response). Your tests are no longer dependent on APIs you don't control. If ES or your pagination gem update their code, simply re-record the cassette when you know the API is up and working. There really isn't any other option without making your tests extremely brittle or stubbing things you shouldn't stub.

Note that although we have stubbed tire above - and we don't own it - it's ok in these cases because the return values are entirely irrelevant to the test.