Dynamic RSpec spec generation

Question

I'd like to generate a series of specs dynamically based on an external data source. Specifically, I have a Google Spreadsheet that's downloaded, and each row in the spreadsheet should be used for its own test case:

describe "Cases" do
  before(:all) do
    # Download spreadsheet and
    # populate cases in DB
  end

  Cases.each do |case|
    it "Case #{case.num}" do
      # spec
    end
  end
end

This doesn't work, because, for starters with RSpec, doesn't (as I understand it) 'see' any specs at compile-time, so before(:all) is never executed. If I were to place an empty it block it'd get the before(:all) to execute, but then I'd run into the problem that Cases.each is evaluated before anything else, which is empty because it hasn't been populated yet by the before(:all) block.

In short, I'm confused, and my understanding of RSpec appears very limited. I'd like to fetch data, use that data to set up a bunch of specs, and then have them run by RSpec. This would (might?) work if Cases were an array set up before-hand (outside the describe block), but I need it to be set up at runtime. Is what I want possible to do in RSpec?

Answer 1

This doesn't work, because ... RSpec doesn't ... 'see' any specs at compile-time ...

I was wondering why, and did some research. If you examine the following code and the output, you can draw some conclusions :

• there is no compile time in Ruby
• the body of a class/module is immediately evaluated and executed
• so is the describe method
• the body of a describe block is immediately processed by RSpec
• RSpec stores before blocks and it blocks for later execution
• when the parsing of a file is terminated, RSpec starts reporting the describe strings and executing the stored before/it example blocks

The following code demonstrates this.

module SO
    puts '>>>>> Ruby sees module SO and executes/evaluates its body'
    cases = [1,2,3]

    describe "SO Cases" do
        puts "in module SO, RSpec sees describe Cases self=#{self}"
        before(:each) do
            puts '  in before(:all)'
        end

        cases.each do |case_|
            puts "    in each loop with case_=#{case_}"
            it "Case #{case_}" do
                puts "        check spec for case #{case_}"
            end
        end
    end
end

module M # to avoid "warning: class variable access from toplevel"
    puts '>>>>> Ruby sees module M and executes/evaluates its body'
    describe "M Cases" do
        puts "in module M, RSpec sees describe Cases self=#{self}, ancestors :"
        ancestors.each {|a| puts "    #{a}"}
        print 'self.methods.grep(/^it/) : '; p self.methods.grep(/^it/).sort
        before(:all) do
            puts "  in before(:all) self=#{self}"
            @@cases = [1,2,3]
            puts "  ... now cases=#{@@cases}"
            File.open('generated_cases.rb', 'w') do |fgen|
                fgen.puts 'puts "\n*** inside generated_cases.rb ***"'
                fgen.puts 'module GenSpecs'
                fgen.puts "puts '>>>>> Ruby sees module GenSpecs and executes/evaluates its body'"
                fgen.puts '    describe "GenSpecs Cases" do'
                fgen.puts '        puts "in module GenSpecs, RSpec sees describe Cases"'
                @@cases.each do |case_|
                    puts "    in each loop with case_=#{case_}"
                    fgen.puts <<-IT_SPECS
        it "Case #{case_}" do
            puts "    some spec for case_=#{case_}"
        end
        IT_SPECS
                end
                fgen.puts '    end'
                fgen.puts 'end # module GenSpecs'
                fgen.puts "puts '>>>>> end of ruby file generated_cases.rb <<<<<'"
            end
            puts 'file generated_cases.rb has been closed'
            require_relative 'generated_cases'
        end

        it 'has loaded Cases' do
            @@cases.should_not be_empty
        end
    end
end
puts '>>>>> end of ruby file t2_spec.rb <<<<<'

Execution :

$ ruby -v
ruby 1.9.2p320 (2012-04-20 revision 35421) [x86_64-darwin12.2.0]
$ rspec --version
2.12.2
$ rspec --format nested t2_spec.rb 
>>>>> Ruby sees module SO and executes/evaluates its body
in module SO, RSpec sees describe Cases self=#<Class:0x007fcaf49a6e80>
    in each loop with case_=1
    in each loop with case_=2
    in each loop with case_=3
>>>>> Ruby sees module M and executes/evaluates its body
in module M, RSpec sees describe Cases self=#<Class:0x007fcaf2852e28>, ancestors :
    #<Class:0x007fcaf2852e28>
    RSpec::Core::ExampleGroup
    ...
self.methods.grep(/^it/) : [:it, :it_behaves_like, :it_should_behave_like, :its]
>>>>> end of ruby file t2_spec.rb <<<<<

SO Cases
  in before(:all)
        check spec for case 1
  Case 1
  in before(:all)
        check spec for case 2
  Case 2
  in before(:all)
        check spec for case 3
  Case 3

M Cases
  in before(:all) self=#<RSpec::Core::ExampleGroup::Nested_2:0x007fcaf2836ca0>
  ... now cases=[1, 2, 3]
    in each loop with case_=1
    in each loop with case_=2
    in each loop with case_=3
file generated_cases.rb has been closed

*** inside generated_cases.rb ***
>>>>> Ruby sees module GenSpecs and executes/evaluates its body
in module GenSpecs, RSpec sees describe Cases
>>>>> end of ruby file generated_cases.rb <<<<<
  has loaded Cases

GenSpecs Cases
    some spec for case_=1
  Case 1
    some spec for case_=2
  Case 2
    some spec for case_=3
  Case 3

Finished in 0.01699 seconds
7 examples, 0 failures

Creating a file and requiring it this way is for the demonstration and may not work in your case. I would recommend doing it in two phases : in the first you read the spreadsheet and create a file.rb with a describe and several it examples. In the second phase you launch Ruby to process the generated file.

Answer 2

Remove the before block.

Code in a describe executes when rspec is loaded. It has the disadvantage that it will download and scan the sheet even if these specs aren't going to be run.

The next problem you are going to have to deal with is that the db gets emptied after every spec. So you will need to re-populate the database in a before block for each spec created below.

I would emphasize that I wouldn't recommend this, I would deeply think about having the test cases pulled dynamically. Consider generic tests that prove your code works without the external data. Or pull a copy of the spreadsheet, save it in spec/assets and load it, with more straightforward tests for that particular example.

describe "Cases" do
  # Download spreadsheet and
  # populate cases in DB
  before(:each) do
    # repopulate the DB
  end

  Cases.each do |case|
    it "Case #{case.num}" do
      # spec
    end
  end

  # clean out the db so that the first executed test is not polluted
end