Is it possible to dynamically run benchmark tests?

Question

I have a few different implementations of an interface, and a variety of factors I want to test them under. End goal being to make a grid of results for different implementations in different situations.

I could write a test for each possible combination, but that gets exhausting:

func Benchmark_ImplA_N100_X300(b *testing.B){
   impl := newImplA(100,300)
   runBenchmark(b,impl)
}

The more combinations I add, the more I have to copy/paste. This gets cumbersome fast.

I would love to do something like:

tests := []testing.InternalBenchmark{}
for _, n := range []int{50,100,500,10000}{
   for _,x := range []int{300,200}{
      for name, gen := range implementations{
         impl = gen(n,x)
         bm := testing.InternalBenchmark{Name: fmt.Sprint(name,x,n)}
         bm.F = func(b *testing.B){
            runBench(b,impl)
         }
         tests = append(tests,bm)
      }
   }
}
testing.RunBenchmarks(anything, tests)

so that I can update the list(s) of combinations, and benchmarks will magically run in all combinations. I have tries something like this in main and in TestMain, and nothing ever outputs. I am not sure if I am using it wrong, or if the testing package is just doing something funny.

I really don't care if the go test tool can handle it or if there is some other way.

Answer 1

Yes, it's possible. In your test file (xxx_test.go) create your own TestMain() function, and inside that after assembling the dynamic benchmark cases (values of struct testing.InternalBenchmark), call testing.Main() which properly parses command line flags, creates and sets up testing.M, and prepares and calls testing.RunBenchmarks(). This way your dynamic benchmarks will still be runnable by go test.

Notes: testing.Main() will never return as it calls os.Exit(). If you want to perform further logging and calculations on the benchmark results, you may also call testing.MainStart().Run() (which is what testing.Main() does), and you may pass the exit code which is returned by M.Run() to os.Exit().

Below comes a complete test file which you can simply run with go test -bench ..

The output is: benchmark results of dynamically generated tests (of different implementations with different parameters):

testing: warning: no tests to run
PASS
main.EngineA[impl=0, n=50, x=300]-4       100000             16716 ns/op
main.EngineB[impl=1, n=50, x=300]-4       100000             24788 ns/op
main.EngineA[impl=0, n=50, x=200]-4       100000             10764 ns/op
main.EngineB[impl=1, n=50, x=200]-4       100000             16415 ns/op
main.EngineA[impl=0, n=100, x=300]-4       50000             33426 ns/op
main.EngineB[impl=1, n=100, x=300]-4       30000             48466 ns/op
main.EngineA[impl=0, n=100, x=200]-4       50000             20452 ns/op
main.EngineB[impl=1, n=100, x=200]-4       50000             33134 ns/op
main.EngineA[impl=0, n=500, x=300]-4       10000            163087 ns/op
main.EngineB[impl=1, n=500, x=300]-4        5000            238043 ns/op
main.EngineA[impl=0, n=500, x=200]-4       10000            102662 ns/op
main.EngineB[impl=1, n=500, x=200]-4       10000            163113 ns/op
main.EngineA[impl=0, n=1000, x=300]-4       5000            319744 ns/op
main.EngineB[impl=1, n=1000, x=300]-4       3000            512077 ns/op
main.EngineA[impl=0, n=1000, x=200]-4      10000            201036 ns/op
main.EngineB[impl=1, n=1000, x=200]-4       5000            325714 ns/op
ok      _/xxx/src/play  27.307s

And the source (a test file, e.g. dynbench_test.go):

package main

import (
    "fmt"
    "testing"
)

type Engine interface {
    Calc()
}

type EngineA struct{ n, x int }

func (e EngineA) Calc() {
    for i := 0; i < e.n; i++ {
        a, b := make([]byte, e.x), make([]byte, e.x)
        copy(b, a)
    }
}

type EngineB struct{ n, x int }

func (e EngineB) Calc() {
    for i := 0; i < e.n*2; i++ {
        a, b := make([]byte, e.x/2), make([]byte, e.x/2)
        copy(b, a)
    }
}

func TestMain(m *testing.M) {
    implementations := [](func(n, x int) Engine){
        func(n, x int) Engine { return EngineA{n, x} },
        func(n, x int) Engine { return EngineB{n, x} },
    }

    benchmarks := []testing.InternalBenchmark{}
    for _, n := range []int{50, 100, 500, 1000} {
        for _, x := range []int{300, 200} {
            for name, gen := range implementations {
                impl := gen(n, x)
                bm := testing.InternalBenchmark{
                    Name: fmt.Sprintf("%T[impl=%d, n=%d, x=%d]", impl, name, n, x)}
                bm.F = func(b *testing.B) {
                    for i := 0; i < b.N; i++ {
                        impl.Calc()
                    }
                }
                benchmarks = append(benchmarks, bm)
            }
        }
    }
    anything := func(pat, str string) (bool, error) { return true, nil }

    testing.Main(anything, nil, benchmarks, nil)
}

Notes #2:

testing.Main(), testing.MainStart() and testing.InternalBenchmark may change (or get removed) in a future release of Go:

An internal function / internal type but exported because it is cross-package; part of or called by the implementation of the "go test" command.

Answer 2

I suppose reading the docs would help. I overlooked

func Benchmark(f func(b *B)) BenchmarkResult

which will run my function wothout need for any test harness at all.

Benchmark benchmarks a single function. Useful for creating custom benchmarks that do not use the "go test" command.

That way I can iterate through my test cases and create a function for each possibility, then run the benchmark directly.