Testing: how to focus on behavior instead of implementation without losing speed?

Question

It seems, that there are two totally different approaches to testing, and I would like to cite both of them.

The thing is, that those opinions were stated 5 years ago (2007), and I am interested, what has changed since then and which way should I go.

Brandon Keepers:

The theory is that tests are supposed to be agnostic of the implementation. This leads to less brittle tests and actually tests the outcome (or behavior).

With RSpec, I feel like the common approach of completely mocking your models to test your controllers ends up forcing you to look too much into the implementation of your controller.

This by itself is not too bad, but the problem is that it peers too much into the controller to dictate how the model is used. Why does it matter if my controller calls Thing.new? What if my controller decides to take the Thing.create! and rescue route? What if my model has a special initializer method, like Thing.build_with_foo? My spec for behavior should not fail if I change the implementation.

This problem gets even worse when you have nested resources and are creating multiple models per controller. Some of my setup methods end up being 15 or more lines long and VERY fragile.

RSpec’s intention is to completely isolate your controller logic from your models, which sounds good in theory, but almost runs against the grain for an integrated stack like Rails. Especially if you practice the skinny controller/fat model discipline, the amount of logic in the controller becomes very small, and the setup becomes huge.

So what’s a BDD-wannabe to do? Taking a step back, the behavior that I really want to test is not that my controller calls Thing.new, but that given parameters X, it creates a new thing and redirects to it.

David Chelimsky:

It’s all about trade-offs.

The fact that AR chooses inheritance rather than delegation puts us in a testing bind – we have to be coupled to the database OR we have to be more intimate with the implementation. We accept this design choice because we reap benefits in expressiveness and DRY-ness.

In grappling with the dilemma, I chose faster tests at the cost of slightly more brittle. You’re choosing less brittle tests at the cost of them running slightly slower. It’s a trade-off either way.

In practice, I run the tests hundreds, if not thousands, of times a day (I use autotest and take very granular steps) and I change whether I use “new” or “create” almost never. Also due to granular steps, new models that appear are quite volatile at first. The valid_thing_attrs approach minimizes the pain from this a bit, but it still means that every new required field means that I have to change valid_thing_attrs.

But if your approach is working for you in practice, then its good! In fact, I’d strongly recommend that you publish a plugin with generators that produce the examples the way you like them. I’m sure that a lot of people would benefit from that.

Ryan Bates:

Out of curiosity, how often do you use mocks in your tests/specs? Perhaps I'm doing something wrong, but I'm finding it severely limiting. Since switching to rSpec over a month ago, I've been doing what they recommend in the docs where the controller and view layers do not hit the database at all and the models are completely mocked out. This gives you a nice speed boost and makes some things easier, but I'm finding the cons of doing this far outweigh the pros. Since using mocks, my specs have turned into a maintenance nightmare. Specs are meant to test the behavior, not the implementation. I don't care if a method was called I just want to make sure the resulting output is correct. Because mocking makes specs picky about the implementation, it makes simple refactorings (that don't change the behavior) impossible to do without having to constantly go back and "fix" the specs. I'm very opinionated about what a spec/tests should cover. A test should only break when the app breaks. This is one reason why I hardly test the view layer because I find it too rigid. It often leads to tests breaking without the app breaking when changing little things in the view. I'm finding the same problem with mocks. On top of all this, I just realized today that mocking/stubbing a class method (sometimes) sticks around between specs. Specs should be self contained and not influenced by other specs. This breaks that rule and leads to tricky bugs. What have I learned from all this? Be careful where you use mocking. Stubbing is not as bad, but still has some of the same issues.

I took the past few hours and removed nearly all mocks from my specs. I also merged the controller and view specs into one using "integrate_views" in the controller spec. I am also loading all fixtures for each controller spec so there's some test data to fill the views. The end result? My specs are shorter, simpler, more consistent, less rigid, and they test the entire stack together (model, view, controller) so no bugs can slip through the cracks. I'm not saying this is the "right" way for everyone. If your project requires a very strict spec case then it may not be for you, but in my case this is worlds better than what I had before using mocks. I still think stubbing is a good solution in a few spots so I'm still doing that.

Answer 1

I think all three opinions are still completely valid. Ryan and I were struggling with the maintainability of mocking, while David felt the maintenance tradeoff was worth it for the increase in speed.

But these tradeoffs are symptoms of a deeper problem, which David alluded to in 2007: ActiveRecord. The design of ActiveRecord encourages you to create god objects that do too much, know too much about the rest of the system, and have too much surface area. This leads to tests that have too much to test, know too much about the rest of the system, and are either too slow or brittle.

So what's the solution? Separate as much of your application from the framework as possible. Write lots of small classes that model your domain and don't inherit from anything. Each object should have limited surface area (no more than a few methods) and explicit dependencies passed in through the constructor.

With this approach, I've only been writing two types of tests: isolated unit tests, and full-stack system tests. In the isolation tests, I mock or stub everything that is not the object under test. These tests are insanely fast and often don't even require loading the whole Rails environment. The full stack tests exercise the whole system. They are painfully slow and give useless feedback when they fail. I write as few as necessary, but enough to give me confidence that all my well-tested objects integrate well.

Unfortunately, I can't point you to an example project that does this well (yet). I talk a little about it in my presentation on Why Our Code Smells, watch Corey Haines' presentation on Fast Rails Tests, and I highly recommend reading Growing Object Oriented Software Guided by Tests.