Strategies for gem tests to ensure the gem works with Rails 3.x and 4.0?

Question

I've seen a few examples of dummy Rails apps (for testing, so they live under test or spec dirs, typically) for use with the Appraisals gem that supposedly work with both Rails 3.x and Rails 4, but they seem hackish and not fully functional. It is somewhat expected, as it is a stripped down Frankenstein monster that is trying to be compatible with various versions of Rails 3 as well as Rails 4.

I've referred to projects that attempt to do this sort of testing (as of late March 2013) like less-rails and ember-rails, but this way to test with various version of Rails doesn't seem very clean, and it is non-trivial to try to debug a non-standard Rails application, especially in a beta version of Rails.

It would be great to have a cleaner way to test that allows you to have a full Rails application for each version of Rails to test with that through some magic is not that difficult to setup or maintain and don't require non-standard path hacks in places, etc.

What are the available strategies for testing gems with various versions of Rails (including at least latest Rails 3.1.x, 3.2.x, and 4.0.0.beta1), and what are the pros and cons of each?

Answer 1

A few options from the related thread on the rails-core list:

Option 1: Appraisal gem and single Rails dummy app

Ken Collins mentioned using appraisal and a Rails "dummy" app:

I test minitest-spec-rails against 3.0, 3.1, 3,2 and 4.0 using a mix of appraisal and dummy_app that minimally configures itself depending which rails version it is testing against. Some links:

https://github.com/metaskills/minitest-spec-rails https://github.com/metaskills/minitest-spec-rails/blob/master/test/dummy_app/init.rb

Similar techniques are used in less-rails, ember-rails, and high_voltage among others.

I used a similar setup to high_voltage in restful_json (v3.3.0) but with a full Rails app created with 4.0.0-beta1 that I modified minimally to also work with Rails 3.1.x/3.2.x.

Update: May want to see permitters for more recent example.

Pros: Fairly simple. Can test against various Rails versions from command-line, etc. Can be very minimal Rails app configuration, or can use full Rails app with minor differences.

Cons: Still reusing same Rails app for multiple Rails versions, so some conditionals and unneeded config. (Possible issues with some files not being applicable in another version of Rails, etc. but does not appear to be a big problem.)

Option 2: Rails version as environment variable with single Gemfile, single Rails dummy app, relying on travis-ci to test in multiple versions

Steve Klabnik mentioned a solution that works with a single Gemfile, a single full Rails app (even though under "dummy" dir, and no use of the appraisal gem, by relying on travis-ci to test:

I've been meaning to discuss this topic more, as I've been doing it for a bunch of my gems lately. I have two that do this:

Draper: https://github.com/drapergem/draper

LocaleSetter: https://github.com/jcasimir/locale_setter/

Basically, I embed an entire Rails application into the gem, and then run it against multiple versions of Rails on travis via env vars.

Pros: Simple. No dependency on appraisal gem (not that it is a problem, but may be easier to maintain).

Cons: Still reusing same Rails app for multiple Rails versions from what I can tell. Unless using travis-ci or something that starts with a clean gemset (i.e. if running at command-line), not currently differentiating gemsets so newer gem may be used with older Rails, etc., but Steve said if that were to cause a problem, you could just blow away the lock and re-bundle.

Answer 2

There's a third option : using multiple gemfiles and multiple dummy apps.

Gemfiles

Bundler has an useful option named --gemfile. With it, you can specify what file to use instead of Gemfile, and it will generate a lock file after the same name :

bundle install --gemfile Gemfile.rails3
bundle install --gemfile Gemfile.rails4

This will generate Gemfile.rails3.lock and Gemfile.rails4.lock. So, those Gemfiles can be a copy of your main Gemfile forcing rails version :

source "http://rubygems.org"
gemspec
gem "jquery-rails"
gem "rails", '~>4' 

Using gemfiles from dummy apps

Then you have two dummy apps, one for rails-3 and one for rails-4. To use their proper gemfile while running (for example) migrations :

cd test/dummy_rails3
BUNDLE_GEMFILE=../../Gemfile.rails3 bundle exec rake db:migrate
cd ../dummy_rails4
BUNDLE_GEMFILE=../../Gemfile.rails4 bundle exec rake db:migrate

Yeah, that's probably the worst part. But this is mostly a one time setup.

Using gemfiles from rake

To instruct which version to use while running tests, set the environment variable BUNDLE_GEMFILE in Rakefile :

#!/usr/bin/env rake

rails_version = ENV[ 'RAILS_VERSION' ] || '4'

if rails_version == '3'
  ENV[ 'BUNDLE_GEMFILE' ] = 'Gemfile.rails3'
else
  ENV[ 'BUNDLE_GEMFILE' ] = 'Gemfile.rails4'
end

begin
  require 'bundler/setup'
rescue LoadError
  puts 'You must `gem install bundler` and `bundle install` to run rake tasks'
end

I prefer to ask the user to pass RAILS_VERSION instead of directly BUNDLE_GEMFILE because it's easier to remember and we can just pass "3" or "4".

Using correct dummy app from tests

Finally, in test_helper, switch the dummy app depending on what rails version has been asked for :

# Configure Rails Environment
ENV["RAILS_ENV"] = "test"

dummy_app = ENV[ 'RAILS_VERSION' ] == '3' ? 'dummy_rails3' : 'dummy_rails4'

require File.expand_path("../#{dummy_app}/config/environment.rb",  __FILE__)
require "rails/test_help"

From your user perspective

For your user to run tests, he will have to do a one time setup by running the migration tasks with BUNDLE_GEMFILE, which is not that sexy.

But once done, user can run tests against rails-3 and rails-4 without the need to generate the Gemfile each time he wants to switch version, and you can have version specific code and configuration within your test apps without having to put if Rails.version >= '4' statements everywhere.

To run specs :

RAILS_VERSION=3 bundle exec rake test
bundle exec rake test # rails-4 is the default in code I wrote

You can see an example for this method in my activerecord_any_of gem.