How can I benchmark code that mutates the setup data?

Question

The current implementation of the built-in benchmarking tool appears to run the code inside the iter call multiple times for each time the setup code outside the iter is run. When the code being benchmarked modifies the setup data, subsequent iterations of the benchmarked code are no longer benchmarking the same thing.

As a concrete example, I am benchmarking how fast it takes to remove values from a Vec:

#![feature(test)]

extern crate test;

use test::Bencher;

#[bench]
fn clearing_a_vector(b: &mut Bencher) {
    let mut things = vec![1];
    b.iter(|| {
        assert!(!things.is_empty());
        things.clear();
    });
}

This will fail:

test clearing_a_vector ... thread 'main' panicked at 'assertion failed: !things.is_empty()', src/lib.rs:11

Performing a similar benchmark of pushing an element onto the vector shows that the iter closure was executed nearly 980 million times (depending on how fast the closure is). The results could be very misleading if there's a single run that does what I expect and millions more that don't.

Tests were run with Rust nightly 1.19.0 (f89d8d184 2017-05-30)

Answer 1

For all people that still have this problem, there is a pretty good solution nowadays: The (well-maintained) framework / crate Criterion provides a timing loop with two closures: The first one sets up the initial state that's given to the second one (which is timed). Essentially, you can just clone the original state in the first closure and return that to the second one, so the original state is used and only the actual function is benchmarked.

This looks about like this:

for input in input_queue {
    c.bench_function("Calculation", |b| {
        b.iter_batched(
            || black_box(original_state.clone()),
            |mut original_state| {
                original_state.calculate(black_box(input));
            },
            BatchSize::PerIteration,
        )
    });
}

More information about this is is available at the official documentation.