How long is too long for a test suite? [closed]

Question

I'm creating a website on rails and have so far been using travis-ci for continuous integration. I'm trying out wercker as well for continuous integration and deployment. The testing step failed on wercker because it ran for more than 25 minutes and it occured to me that maybe my tests take abnormally long for some reason.

The website I'm working on is sizeable, but not massive. I test using rspec and capybara, using webkit for integration tests. I write quite thorough integration tests, trying to make sure I cover each feature. On travis-ci, the entire process takes 25-30 minutes to run (including installing the bundle).

This might be a vague question for this forum, but nevertheless I'd like to get some inputs. Is it unacceptable to have test suites that run for half an hour or more? What test suite times do you normally experience for a set of integration tests?

Answer 1

It is normal for commercial web sites to have integration test suites, even well-designed ones, which would take an hour or more to run if they were run in a single process on a developer machine. So you've given us no reason to think that you're writing too many tests or writing them so that they run unusually slowly. However, that's much too long to wait to learn whether your commit was good. In my experience half an hour is too long and 15' is marginal; if it takes that long to run all the tests the person who triggered the build will start something else or wander away while the build is running and then will have to context-switch or won't be around when the build breaks. Longer builds also increase the average number of commits in a given build, which makes it harder to assign blame when the build breaks.

So get your CI build to run as fast as it reasonably can. Big topic, but a few starting points:

• The parallel_tests gem is the way to run your suite (both unit tests and Cucumber) as fast as possible on a single box (which will only get you so far, but maybe that's far enough for now).
• Here's another gem (that I haven't used) for splitting Cucumber scenarios across boxes: https://github.com/cloudcastle/cucumber_in_groups
• Travis CI, CircleCI and presumably other hosted continuous integration services provide ways to split your tests across boxes.

It is also helpful to have a unit test suite that covers most or all of your code and runs much faster than your integration tests (in seconds or at most a few minutes) so that most errors are caught before the integration tests run.