End-to-end testing a RESTful Web service (Rails)

Question

I'm trying to sift through the myriad of test solutions out there, and I'm not even sure if I'm headed the right direction. The story is: we're running a RESTful Web service, implemented as a Rails app, which backs our mobile clients. We're unit testing the Web service (of course), but that involves mocking out many parts of the application, for instance, the search stack (Apache SOLR).

Moreover, our tests don't (i.e. cannot!) cover critical routes such as the mobile sign-in/sign-on process, since that involves communication between the API application and the mobile website, where the user can enter credentials, e.g. for SSO (Janrain Engage). Hence, a standard Rails integration test won't do.

I realize that in theory, if the test suite is very well designed, where mocking only happens strictly at the those join points where the tests for the next layer start, then by unit or functional testing the service API and the mobile website separately, one could get the same test coverage. I find that in practice though, this is an illusion if you have several developers working on the test suite independently; I just admit that our unit tests are simply not that well designed. Particularly when exercising TDD I found that while the tests can drive the application code, the test code design is only tailored to the unit under test, resulting in a rather wildly grown test suite.

Another thing I found is that sometimes we didn't detect regressions purely using unit tests, where e.g. bad queries were sent to the SOLR server due to knock-on effects. That's why I thought the only true way to ensure that the entire stack works along the critical routes is to automatically end-to-end test it on a staging server before every deployment, i.e. having actual HTTP requests sent to the app.

My questions would be:

• do you think this is a reasonable thing to do at all? I found very little information about end-to-end testing a live API on the Web, leaving me wondering whether I'm making any sense at all
• which tools/setup would you suggest? We use Watir to run acceptance tests for our website, but it seems to be overkill for a Web service (no browser environment needed, no JS or anything UI-ish). Something as simple as a Ruby script even?
• any general best practices or advice you can give me w.r.t. to designing such tests?

Answer 1

This might be of interest to you: http://groups.google.com/group/ruby-capybara/browse_thread/thread/5c27bf866eb7fad3

What you might want to to try is combining Cucumber (or similar) with one of the tools mentioned in the link above, so you can do something like

Given I have 2 posts
When I send "DELETE" to post with id 1
Then I should have 1 post

This way you can test the API's full stack and check the result.

Answer 2

We've faced similar problems lately, like testing only on the unit level was fast but not enough to spot integration related anomalies.

I really like Cucumber's gherkin syntax, the value for me in it is that we can run the exercise once and then make several expectations after with separate titles, like if they would be separate RSpec it blocks.

There is a gem called Turnip, which makes possible to write RSpec tests with the gherkin syntax. We have been using it for a while and I'm quite happy with it, we even started having less integration level defects.

I collected my experience regarding in a blog post: https://blog.kalina.tech/2019/10/end-to-end-testing-for-api-only-rails-apps-with-cucumber-syntax.html