Run a benchmark in parallel, i.e. simulate simultaneous requests

Question

When testing a database procedure invoked from an API, when it runs sequentially, it seems to run consistently within ~3s. However we've noticed that when several requests come in at the same time, this can take much longer, causing time outs. I am trying to reproduce the "several requests at one time" case as a go test.

I tried the -parallel 10 go test flag, but the timings were the same at ~28s.

Is there something wrong with my benchmark function?

func Benchmark_RealCreate(b *testing.B) {
    b.ResetTimer()
    for n := 0; n < b.N; n++ {
        name := randomdata.SillyName()
        r := gofight.New()
        u := []unit{unit{MefeUnitID: name, MefeCreatorUserID: "user", BzfeCreatorUserID: 55, ClassificationID: 2, UnitName: name, UnitDescriptionDetails: "Up on the hills and testing"}}
        uJSON, _ := json.Marshal(u)
        r.POST("/create").
            SetBody(string(uJSON)).
            Run(h.BasicEngine(), func(r gofight.HTTPResponse, rq gofight.HTTPRequest) {
                assert.Contains(b, r.Body.String(), name)
                assert.Equal(b, http.StatusOK, r.Code)
            })
    }
}

Else how I can achieve what I am after?

Answer 1

The -parallel flag is not for running the same test or benchmark parallel, in multiple instances.

Quoting from Command go: Testing flags:

-parallel n
    Allow parallel execution of test functions that call t.Parallel.
    The value of this flag is the maximum number of tests to run
    simultaneously; by default, it is set to the value of GOMAXPROCS.
    Note that -parallel only applies within a single test binary.
    The 'go test' command may run tests for different packages
    in parallel as well, according to the setting of the -p flag
    (see 'go help build').

So basically if your tests allow, you can use -parallel to run multiple distinct testing or benchmark functions parallel, but not the same one in multiple instances.

In general, running multiple benchmark functions parallel defeats the purpose of benchmarking a function, because running it parallel in multiple instances usually distorts the benchmarking.

However, in your case code efficiency is not what you want to measure, you want to measure an external service. So go's built-in testing and benchmarking facilities are not really suitable.

Of course we could still use the convenience of having this "benchmark" run automatically when our other tests and benchmarks run, but you should not force this into the conventional benchmarking framework.

First thing that comes to mind is to use a for loop to launch n goroutines which all attempt to call the testable service. One problem with this is that this only ensures n concurrent goroutines at the start, because as the calls start to complete, there will be less and less concurrency for the remaining ones.

To overcome this and truly test n concurrent calls, you should have a worker pool with n workers, and continously feed jobs to this worker pool, making sure there will be n concurrent service calls at all times. For a worker pool implementation, see Is this an idiomatic worker thread pool in Go?

So all in all, fire up a worker pool with n workers, have a goroutine send jobs to it for an arbitrary time (e.g. for 30 seconds or 1 minute), and measure (count) the completed jobs. The benchmark result will be a simple division.

Also note that for solely testing purposes, a worker pool might not even be needed. You can just use a loop to launch n goroutines, but make sure each started goroutine keeps calling the service and not return after a single call.

Answer 2

I'm new to go, but why don't you try to make a function and run it using the standard parallel test?

func Benchmark_YourFunc(b *testing.B) {
    b.RunParralel(func(pb *testing.PB) {
        for pb.Next() {
            YourFunc(staff ...T)
        }
    })
}