Automated performance testing framework

Question

Is there a framework to write automated performance tests on the JVM ?

I would like to set some performance targets and flash a red light (or rather print a red console message) everytime the perfomance is below the goal.

I have some benchmarks written with Calliper that I run manually, however I would like the performance test suite, to be run automatically as I do for unit-testing.

No databases, or web server.

Answer 1

Since 4-th version JUnit supports that functionality:

import org.junit.*;

public class JunitTest4 {

    // timings below are in milliseconds
    @Test(timeout = 1000)  
    public void infinity() {
        // will fail    
        while (true);  
    }  

}

But sadly, it just a basic functionality, not sure you'll be completely pleased: you can't measure something less than millisecond, you can't store history line for tests, you can't see how far you're from the goal.

Answer 2

I'm not a performance testing specialist, so I can't judge how difficult this would be to implement, but it should be possible with JUnit Rules.

It would work similar to the @Test annotation mentioned by om-nom-nom, but you would provide a rule which for example runs each test 100 times for warm up than runs it 100 times and calculates the stdev and fails the test when average time is 2 stdevs larger then the set timeoiut. ... Or whatever the rules should be.

Here is an article to JUnit Rules.

Basically the same approach should work with ScalaTest