rspec/capybara/poltergeist tests pass 100% locally, random tests fail on teamcity CI server

Question

I have a bunch of rspec tests and roughly 30 of those are headless tests through capybara/poltergeist. Every time i run the tests locally, all 100% of the tests pass.

I recently set up a teamcity continuous integration server and the test results have been extremely inconsistent.

I ran the tests 10 times, without doing any changes between the tests. Only 1 of the rounds passed 100%. The others had 1-2 failures (most of them weren't the same), all relating to the headless browser testing. Here is an example of one of the failures:

ActionView::Template::Error: Couldn't find Spree::Address with id=1072978592

Stack trace:
./app/models/spree/order_decorator.rb:50:in `initialize_default_address'
./app/models/spree/order_decorator.rb:42:in `initialize_shipment'
./app/views/layouts/core.html.erb:23:in `_8cb98e121af585621c1d08e3ec1f6022'
./app/views/layouts/default.html.erb:14:in `_588b3208edc213a939dffd2ad73f4f26'

This failure is odd because i stubbed the function that looks for an address with an ID and returns a factorygirl model. I do not have this problem locally.

Here is another error from one of the test runs:

Capybara::ExpectationNotMet: expected to find link "10" but there were no matches

Stack trace:
./spec/features/simply_ship_spec.rb:102:in `block (2 levels) in <top (required)>'

Again, when i run this locally, this error doesn't happen, and when i switch to selenium and watch the test, the link is clearly there.

Both of these errors (not all, but a couple examples) happened individually, on different runs, without doing any updates to the project. Does anybody have any ideas why this is happening?

rspec-rails 2.14.0 capybara 2.1 poltergeist 1.3.0 rails 3.2.14 ruby 1.9.3 p448

Answer 1

You may try capybara-screenshot gem to clarify page state during fails.

Answer 2

It's hard to say without seeing your code, but it sounds like you may have race conditions in your code. If your script is running faster than the application code, then you'll get occasional failures like this.

You may look into where these failures are happening and making sure the application is caught up before continuing.