automatically prevent performance regressions

Question

Whenever I have a bug, my first piece of work is to write a failing test that shows that the bug exists. When added to my automated test suite, this test serves to ensure that the bug doesn't appear again.

With performance bugs, however, I'm unable to do this. Any assertion of performance would only be correct on my own machine, and not appropriate to check in to an automated test suite. This takes my usual tool for preventing regression out of my bag.

How do you prevent performance regressions in your own code?

An ideal answer:

• Is language agnostic.
• Prevents regressions from getting shipped, automatically.
• Provides the usual dev cycle: checkout, patch, regression-test, checkin.
• Works for open-source projects, where devs don't necessarily know the "usual" performance.

Answer 1

Great question, tough problem.

As you have indicated in your question, the crux of the problem is the differences in environments i.e. between your development box and the test/QA environment.

To mitigate against performance regressions you could set-up a continuous deployment environment e.g. as well as automatically deploying to a test environment, you should also deploy to a QA environment which has operating parameters as similar to the final Production environment as possible (e.g. similar hardware, similar sized data set, etc.). This is commonly known as "Build once, Deploy many"

With this continuous deployment system in place you should also include acceptance tests to check for performance constraints e.g. the system should respond to a request within x milliseconds/seconds, a common non-functional requirement (NFR). When your code is then checked-in, it would be then tested automatically against these environments so you can quickly see when performance issues arise.

To help debug these performance issues you'd ideally also record performance metrics whilst running the tests. These test might then give you a good indicator as to where the problem may be manifesting itself e.g. you'll be able to see that a particular method is taking the majority of time to complete.

This set-up is difficult to establish in practice, but it would definitely allow you to "fail fast" and detect performance issues quickly.

Answer 2

I have used two different approaches to this sort of problem.

1. As @Fresh suggested monitoring performance on a realistic, possibly live, environment is a good option. That avoids both false positive and false negative results you might see on a development environment. That may have to occur post-commit but it is also the only tool I can think of to try to catch interactions between system components which might never show up as a performance issue locally or with different data sets.

2. Some metrics from a development environment which can be run pre-checkin still give useful hints about possible performance problems. In particular I like to have my test suite report on the slowest N tests in each unit/integration/acceptance category. While far from a sure indicator new appearances in those results or an increase in test run times are a good hint of a performance problem. This helps keep the test suites' runtime down as well which is also valuable.