🎓 A1Learner
🔍

Rust Benchmarking with Criterion

Criterion is a benchmarking tool that measures code many times and compares the results. A later run can then say whether the code got faster or slower.

Wire Criterion in

Criterion is a dev-dependency. The bench file needs harness = false. Criterion brings its own main function:

[dev-dependencies]
criterion = "0.5"

[[bench]]
name = "sum_bench"
harness = false

One function, several sizes

A group holds related benchmarks. bench_with_input runs the same function at each size:

let mut group = c.benchmark_group("sum");
for n in [1_000u64, 100_000] {
    group.bench_with_input(BenchmarkId::new("loop", n), &n, |b, &n| {
        b.iter(|| sum_loop(black_box(n)));
    });
}
group.finish();

Put rivals in the same group

Add the second way beside the first. Same sizes. Same conditions. The report then lines the two ways up:

group.bench_with_input(BenchmarkId::new("formula", n), &n, |b, &n| {
    b.iter(|| sum_formula(black_box(n)));
});

Read the report line

One finished run on this machine printed the line below. The three times are a low estimate, the best estimate, and a high estimate:

sum/loop/1000    time:   [7.7398 ns 8.9859 ns 10.100 ns]

Example

use std::time::Instant;

fn fib_slow(n: u64) -> u64 {
    if n < 2 {
        n
    } else {
        fib_slow(n - 1) + fib_slow(n - 2)
    }
}

fn fib_fast(n: u64) -> u64 {
    let mut a = 0u64;
    let mut b = 1u64;
    for _ in 0..n {
        let next = a + b;
        a = b;
        b = next;
    }
    a
}

fn main() {
    let start = Instant::now();
    let slow = fib_slow(30);
    let slow_time = start.elapsed();

    let start = Instant::now();
    let fast = fib_fast(30);
    let fast_time = start.elapsed();

    println!("fib_slow(30) = {slow}");
    println!("fib_fast(30) = {fast}");
    println!("answers match = {}", slow == fast);
    println!("slow took longer = {}", slow_time > fast_time);
}

Key points

  • Criterion measures the same code many times, not once.
  • Benchmark group: a named set of related benchmarks.
  • The group shares settings.
  • Its results print together.
  • Use bench_with_input to run one function over several input sizes.
  • BenchmarkId gives each size its own line in the report.
  • Use black_box on inputs and results.
  • It stops the compiler from deleting the work you meant to measure.
  • Benchmarks live in the benches/ folder.
  • Cargo builds each file as its own programme.
  • cargo bench builds in release mode.
  • Numbers from a debug build would lie about real speed.
  • The next run compares against them.
  • It prints a change line with a percentage.

In this example, two functions compute the same Fibonacci number. fib_slow calls itself twice for every step. fib_fast keeps a running pair and loops. main times each one with the stopwatch from the Benchmarks lesson. Both print 832040, so the answers match. The last line shows the slow one really took longer. One timing still proves little, though. Criterion would run each function thousands of times. It would then report the gap with confidence, not luck.

Note

Criterion saves results under target/criterion.

Try it Yourself

Edit the code, then press Run.

Running is simulated · Rust Playground

Exercise

What do the three times in a Criterion report line mean?