Menu
Michael Feathers, in Working Effectively With Legacy Code, on pages 13-14 mentions:
A unit test that takes 1/10th of a second to run is a slow unit test... If [unit tests] don't run fast, they aren't unit tests.
I can understand why 1/10th a second is too slow if one has 30,000 tests, as it would take close to an hour to run. However, does this mean 1/11th of a second is any better? No, not really (as it's only 5 minutes faster). So a hard fast rule probably isn't perfect.
Thus when considering how slow is too slow for a unit tests, perhaps I should rephrase the question. How long is too long for a developer to wait for the unit test suite to complete?
To give an example of test speeds. Take a look at several MSTest unit test duration timings:
0.2637638 seconds 0.0589954 0.0272193 0.0209824 0.0199389 0.0088322 0.0033815 0.0028137 0.0027601 0.0008775 0.0008171 0.0007351 0.0007147 0.0005898 0.0004937 0.0004624 0.00045 0.0004397 0.0004385 0.0004376 0.0003329 The average for all 21 of these unit tests comes to 0.019785 seconds. Note the slowest test is due to it using Microsoft Moles to mock/isolate the file system.
So with this example, if my unit test suite grows to 10,000 tests, it could take over 3 minutes to run.
957
Still, it seems as though a 10 second short-term attention span is more or less hard-wired into the human brain. Thus, a unit test suite used for TDD should run in less than 10 seconds. If it's slower, you'll be less productive because you'll constantly lose focus.
All unit tests should run in under a second (that is all unit tests combined should run in 1 second).
Tests Should Be Fast That defeats the whole purpose of having a suite of unit tests in the first place, which is to boost the developers' confidence to make changes to the code. The tests can't work as the safety net they're supposed to be if they're not run often.
It's common that slow tests are overloaded—they try to do too many things. By paring down the steps each test has to perform, the overall time for each test goes down, and often there can be a synergistic effect. You may find that running multiple single-purpose tests is quicker than running one overloaded test.
I've looked at one such project where the number of unit tests made the system take too long to test everything. "Too long" meaning that you basically didn't do that as part of your normal development routine.
However, what they had done was to categorize the unit tests into two parts. Critical tests, and "everything else".
Critical tests took just a few seconds to run, and tested only the most critical parts of the system, where "critical" here meant "if something is wrong here, everything is going to be wrong".
Tests that made the entire run take too long was relegated to the "everything else" section, and was only run on the build server.
Whenever someone committed code to the source control repository, the critical tests would again run first, and then a "full run" was scheduled a few minutes into the future. If nobody checked in code during that interval, the full tests was run. Granted, they didn't take 30 minutes, more like 8-10.
This was done using TeamCity, so even if one build agent was busy with the full unit test suit, the other build agents could still pick up normal commits and run the critical unit tests as often as needed.
66
I've only ever worked on projects where the test suite took at least ten minutes to run. The bigger ones, it was more like hours. And we sucked it up and waited, because they were pretty much guaranteed to find at least one problem in anything you threw at them. The projects were that big and hairy.
I wanna know what these projects are that can be tested comprehensively in seconds.
(The secret to getting things done when your project's unit tests take hours is to have four or five things you're working on at the same time. You throw one set of patches at the test suite and you task-switch, and by the time you're done with the thing you switched to, maybe your results have come back.)
34
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
