What is the best way to smoke test a staging/production environment in Rails?

Question

First, the setup...

I am currently developing a Rails 3 application on Mac OS X using Ruby 1.8.7 MRI, running tests and local dev against a MySQL database. I have 3 "other" non-local environments that we use at my company for every application called dev, tqa, and prod. These run in Tomcat using JRuby (1.8.7) with Oracle as the backend.

As you can see, the environments are quite different, and we've run into some bugs on deployment to an Oracle/JRuby environment that don't exist locally (like date handling and specifying default schemas in Oracle).

I love running something like Cucumber/Webrat/Capybara locally to hit every URL exposed in the app to make sure the basic stuff is working (ie; a smoke test). Ideally, it hits every url, and would do some simple things like entering data in forms and clicking on buttons, etc.

Ideally, when I deploy to dev/tqa, I would run something similar, except pointed at the deployed app instead of the local application. Cucumber seems optimized to hit a locally running application and integrates well with Rails, but cannot run against what is for all intents an "external" application (or at least I can't find an easy way that actually works).

Also, when I deploy to prod I'd like a similar suite of smoke tests to run, except it would not change the state of the current production database (ie, would just GET URLs).

Something like Selenium could be used, I guess, but I'd really like to just run a rake task and get back the results like I do with Cucumber.

Is there any Rails/Ruby way to go about doing this, or does everyone else just roll their own solution using wget or Selenium?

A similar question was asked here: Automatically smoke test all webpages in application, after deployment

I'm not sure that the question is exactly what I have in mind, though.

Answer 1

Yes, you can write smoke tests with Cucumber and Capybara and run them against remote servers. I've done it and it works. I've also done curl/wget and the like on some projects, but Cucumber+Capybara allows you to interact with pages (even ones that use Javascript), not just scrape them.

• Capybara's Rack::Test driver doesn't support remote requests; its Javascript drivers do. Whether or not you actually need Javascript to work on the pages you're testing, you'll need to use a Javascript driver. Configure Capybara to use a Javascript driver per the driver's docs and tag your tests @javascript. (I recommend the poltergeist/PhantomJS driver; it's faster than Selenium, gives better errors than capybara-webkit, and is easy to set up.) Bonus: you can do things in your tests that need Javascript, and you'll be smoke-testing your entire stack including Javascript.
• Write your tests so that they either don't need to clean up after themselves or do so in a way that is safe in a production database. They can't use DatabaseCleaner. (To prevent accidents, put the tests in their own project with a Gemfile that doesn't contain DatabaseCleaner.) Since the smoke tests will run against a remote server and therefore can't use transactions to clean up either, they must either not modify the database or must specifically delete only the objects that they create.
• Set Capybara.app_host = "http://your-server.yourco.com"
• Set Capybara.run_server = false (not required, but there's no point running a local server that you won't use)
• If your tests modify the database, set your test database environment to the environment you want to smoke-test.
• Deploy and test.

Answer 2

One way to simply beat on your application to see if it blows up is to grab a list of GET requests from your production web server log file and feed those into a program like curl or wget that will fetch them all as quickly as possible.

Here's a simple example:

#!/usr/bin/env ruby
# rerun

require 'uri'

def extract_from_log(base_uri, log_path)
  File.open(log_path) do |log|
    while (line = log.gets)
      if (line.match(%r{"GET (/\S+) HTTP/\d\.\d"}))
        uri = URI.join(base_uri, $1)
        puts uri.to_s
      end
    end
  end
end

base_uri, log_path = ARGV

if (base_uri and log_path)
  extract_from_log(ARGV[0], ARGV[1])
else
  puts "Usage: #{File.basename($0)} <base_uri> <log_path>"
  exit(-1)
end

Given a standard web log file with lines matching "GET /... HTTP/1.1" you can easily extract the paths, but it's necessary to provide the base URI:

rerun http://example.com/ example.log

This will list out all the URLs found within that log file. You can pipe this to wget for fetching purposes:

rerun http://example.com/ example.log | wget -i -

If you've broken anything, you'll start getting errors in your application, and with a proper notification system you'll be able to catch these and track them down.