Automatically comparing Java profiling results for a single unit test

Question

I want to run a single unit test and collect its "profiling" information: how often every method was called, how many instances of certain class were created, how much time did it take to execute certain method/thread, etc. Then, I want to compare this information with some expected values. Are there any Profilers for Java than let me do this (all this should be done automatically, without any GUI or user interaction, of course)?

This is how I want it to work:

public class MyTest {
  @Test
  public void justTwoCallsToFoo() {
    Profiler.start(Foo.class);
    Foo foo = new Foo();
    foo.someMethodToProfile(); // profiler should collect data here
    assertThat(
      Profiler.getTotalCallsMadeTo(Foo.class, "barMethod"), 
      equalTo(3)
    );
  }
}

Answer 1

Capturing snapshots of profiling information automatically is possible with several Java profilers, you have to add VM parameters to the Java start command to load the profiler and you have to tell the profiler to record data and save a snapshot on exit.

In JProfiler, you could use the "profile" ant task to do this. It takes the profiling settings from a config file that is exported from the JProfiler GUI. You can use the Controller API to start recording and save a snapshot.

Then you have two choices:

1. You can use the jpcompare command line utility or the corresponding "compare" ant task to compare the snapshot against a known baseline. There are several comparisons available. You can export the results as HTML or in a machine-readable format (CSV/XML).

2. You can extract data from the snapshot with the jpexport command line utility or the corresponding "export" ant task. Then you can write your own code to analyze the profiling data from a particular run.

For a limited set of profiling data, it is actually possible to use the JProfiler API to write your own profiler that does something like the Profiler.getTotalCallsMadeTo(Foo.class, "barMethod") in your example. See the api/samples/platform example in a JPofiler installation. You would get the information from the hot spot data.

However, for making assertions on whether a specific method was called I would suggest to take an AOP-based approach. Using a profiler-based approach is appropriate for detecting performance regressions.

Disclaimer: My company develops JProfiler.