Parallel Test Execution hangs indefinitely for rspec

Question

I'm running my specs through parallel_tests on the capybara-webkit driver. I have the following ruby environment:

 ruby -v
 ruby 1.9.3p125 (2012-02-16 revision 34643) [x86_64-darwin11.4.2]

Running through rvm on a gemset which contains the following (truncated for capybara, rails, rspec, and parallel_tests for relevance. If seeing a larger swathe of my gemset would help, please let me know):

 *** LOCAL GEMS ***

 ...
 capybara (1.1.2)
 parallel_tests (0.8.12)
 rails (3.2.11)
 rspec (2.11.0)

When I run my test suit on a single process with rake spec, all of my tests run to completion. However, when runnning through parallel_tests, the following happens:

 8 processes for 220 specs, ~ 27 specs per process

Whereafter the processes will eventually start coming back:

 Finished in 11 minutes 15.76 seconds
 Finished in 11 minutes 28.89 seconds

But, after the first 6 processes come back, parallel_spec will hang indefinitely, never terminate, and never print output for the remaining 2 processes.

I'm on a MacBook Pro running OS X Lion, with a 2.4GHz Intel i7.

So my question is simple: Why is it hanging, how can I debug why its hanging, and how can I stop it from hanging and allow parallel_tests to run to completion?

Answer 1

There is some information missing regarding your rspec configuration and library usage that would probably you get an answer to this. That said, I've seen similar behavior in a multi-threaded environment when running rspec for integration specs.

The advice found on https://github.com/grosser/parallel_tests/wiki looks to be misleading in regard to integration specs. Trying to rely on the transaction strategy of DatabaseCleaner or use_transactional_fixtures is guaranteed to result in deadlocks for any non-blocking database adapter.

Capybara spins up multiple threads for integration specs. When the client and server threads attempt to interact with the same records at the same time you'll often end up with timeouts or deadlocks. Occasionally the deadlock can cause your suite run to hang permanently until it is killed manually.

The most solid configuration I've found to prevent this is a combination of sharing the connection between ActiveRecord instances and judicious use of DatabaseCleaner.

# integration_spec_helper.rb

RSpec.configure do |config|
  config.use_transactional_fixtures = false

  class ActiveRecord::Base
    class_attribute :shared_connection

    def self.connection
      self.shared_connection || retrieve_connection
    end
  end

  config.before do |example|
    ActiveRecord::Base.shared_connection = ActiveRecord::Base.connection

    if Capybara.current_driver == :webkit
      DatabaseCleaner.strategy = :deletion
    else
      DatabaseCleaner.strategy = :transaction
    end

    DatabaseCleaner.start
  end

  config.after do
    DatabaseCleaner.clean
  end
end

Answer 2

In order to debug where your Parallel spec execution is stuck, you need to know how the processes are behaving. This means if one or more processes get stuck and on what spec they do so.

For this you need to do a little hack on the rspec configuration:

# Allows to access hanging spec name and source
# Throw the following command in a terminal whenever a spec is hanging
# $ ps -ef | grep rspec
RSpec.configure do |config|
  config.around :each do |example|
    title  = example.metadata[:full_description]
    source =  example.metadata[:block].source_location.join ":"
    $0 = %{rspec #{source} "#{title}"}
    example.run
  end
end

Then open another terminal and execute:

ps -ef | grep "rspec /"

Initially it will show several processes, something like:

julian    7128  7073 93 10:10 pts/1    00:09:54 rspec /home/julian/Lenda/lenda/spec/models/disclosures/docusign/refinance_loan_estimate_form_spec.rb:17 "RefinanceLoanEstimateForm should be able to be generated from non-completed applications"
julian    7141  7073 93 10:10 pts/1    00:09:50 rspec /home/julian/Lenda/lenda/spec/models/disclosures/disclosure_document_spec.rb:14 "DisclosureDocument retrieves fields signed by a signer"
julian    7147  7073 94 10:10 pts/1    00:09:58 rspec /home/julian/Lenda/lenda/spec/controllers/api/v1/applications_controller_spec.rb:192 "Api::V1::ApplicationsController admin user GET #show with a valid application_id "

You need to monitor these processes until they're not changing anymore. When they finish they disappear, so at the end, you might end with only one or two processes stuck, but it really depends on your test suite.

From there you need to debug separately that test to find the issue.