How to ignore benchmarks when not using nightly?

Question

I have a file with some benchmarks and tests and would like to test against stable, beta and nightly. However, either I don't use the benchmark or stable/beta complain. Is there a way to hide all the benchmark parts when using stable/beta?

As an example the following code from the book:

#![feature(test)]

extern crate test;

pub fn add_two(a: i32) -> i32 {
    a + 2
}

#[cfg(test)]
mod tests {
    use super::*;
    use test::Bencher;

    #[test]
    fn it_works() {
        assert_eq!(4, add_two(2));
    }

    #[bench]
    fn bench_add_two(b: &mut Bencher) {
        b.iter(|| add_two(2));
    }
}

I'm using rustup and would like the same file to work with all the builds, calling something like:

rustup run nightly cargo bench --bin bench --features "bench"
rustup run nightly cargo test --bin bench --features "bench"
rustup run beta cargo test --bin bench
rustup run stable cargo test --bin bench

I was able to hide the #![feature(test)] with #![cfg_attr(feature = "bench", feature(test))]. Can I do something similar to the rest of the benchmark parts? What is a good resource for feature flags?

Answer 1

Is there a way to hide all the benchmark parts when using stable/beta?

Yes, and you can do it automatically using a build script, so it is not necessary to specify --features when executing cargo. In the build script you can detect the version of the Rust compiler and define a feature ("nightly" for example). Then, in the source code you can group your benchmarks and enable them if the feature was defined.

Cargo.toml

[package]
build = "build.rs"

[features]
nightly = []

[build-dependencies]
rustc_version = "0.1.*"

build.rs

extern crate rustc_version;

use rustc_version::{version_meta, Channel};

fn main() {
    if version_meta().channel == Channel::Nightly {
        println!("cargo:rustc-cfg=feature=\"nightly\"");
    }
}

src/lib.rs

#![cfg_attr(all(feature = "nightly", test), feature(test))]

#[cfg(all(feature = "nightly", test))]
extern crate test;

pub fn add_two(a: i32) -> i32 {
    a + 2
}

#[cfg(test)]
mod tests {
    // tests
}

#[cfg(all(feature = "nightly", test))]
mod benchs {
    use test::Bencher;
    // benchs
}

Answer 2

Cargo supports a benches directory for benchmark tests. If you put them in there, you can just never run "cargo bench" on beta/stable, and only run it on nightly.

Answer 3

In my projects, I place benchmarks in a separate module, just like I do for tests. I then create a Cargo feature that enables them. In this excerpt, I used the feature name unstable, but you can use anything you'd like:

Cargo.toml

# ...

[features]
unstable = []

# ...

src/lib.rs

#![cfg_attr(feature = "unstable", feature(test))]

#[cfg(test)]
mod tests {
    #[test]
    fn a_test() {
        assert_eq!(1, 1);
    }
}

#[cfg(all(feature = "unstable", test))]
mod bench {
    extern crate test;
    use self::test::Bencher;

    #[bench]
    fn a_bench(b: &mut Bencher) {
        let z = b.iter(|| {
            test::black_box(|| {
                1 + 1
            })
        });
    }
}

The line #[cfg(all(feature = "unstable", test))] says to only compile the following item if the feature is set and we are compiling in test mode anyway. Likewise, #![cfg_attr(feature = "unstable", feature(test))] only enables the test feature flag when the unstable feature is enabled.

Here's an example in the wild.