How to include google benchmark in continuous integration

Question

I like to include google benchmark tests within out continuous integration environment.

How can I check if the code under test has not become slower than it was before? Do I have to manually process the json output of the benchmark?

Is there an elegant way of accounting for different machines?

Is there an elegant way to check for every build on the same machine the performance? Is there a better way than writing a bash script and compare the output of the actual performance with a reference performance?

There has been a similar question, but no good answer to it.

Is there a way to integrate it into Jenkins.

Answer 1

To integrate Google Benchmark in Jenkins, I used this neat plugin:

https://plugins.jenkins.io/benchmark

I had the tests output .json files

benchmarks.exe --benchmark_out=benchmarktest_output.json

and wrote a custom JSON schema for the plugin to understand the output:

{
  "description": "Google Benchmark JSON schema",
  "failure": { "value": true },
  "type": "object",
  "properties": {
    "benchmarks": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "name": { "type": "name" },
          "real_time": { "type": "result" },
          "cpu_time": { "type": "result" },
          "iterations": { "type": "parameter" }
        }
      }
    }
  }
}

I'm not able to display all the output in the plugin though. E.g., I haven't figured out how to display the time unit together with both real and cpu time, only with one of them at the time.