How to check for failures in an RSpec test suite?

Question

I am experimenting with RSpec and considering a system that would change the random seed only when the test suite passes. I am trying to implement this within an after(:suite) block, which executes in the context of an RSpec::Core::ExampleGroup object.

While RSpec::Core::Example has a method "exception" that allows you to check whether any of the tests have failed, there does not seem to be a similar method on RSpec::Core::ExampleGroup or any accessor for the list of Examples. So, how can I check whether the tests have passed or failed?

I understand that this is possible using a custom formatter that keeps track of whether any tests have failed, but it seems like a bad idea for the formatting process to affect the actual running of the tests.

Answer 1

I poked around in the RSpec source code and figured out that the following would work. Just put this code in spec_helper.rb or some other file that gets loaded when you run the tests:

RSpec.configure do |config|
  config.after(:suite) do
    examples = RSpec.world.filtered_examples.values.flatten
    if examples.none?(&:exception)
      # change the seed here
    end
  end
end

The RSpec.world.filtered_examples hash associates example groups to an array of examples from that group. Rspec has features for filtering out certain examples, and this hash appears to only contain the examples that were actually run.

---

One alternative way you could set up your system would be to check the return code of the rspec process. If it is 0, then all the tests passed and you can change the seed.

In a shell script, you could define a command that changes the seed and run:

rspec && change_seed

If your project has a Rakefile, you could set up something like this:

task "default" => "spec_and_change_seed"

task "spec" do
  sh "rspec spec/my_spec.rb"
end

task "spec_and_change_seed" => "spec" do
  # insert code here to change the file that stores the seed
end

If the specs fail, then rake's "spec" task will fail and it will not go on to change the seed.