๐ŸŽ“ A1Learner
๐Ÿ”

Rust Rayon Compile Errors

A rayon compile error is the compiler refusing code that could race. The compiler guards rayon's one promise of no data races.

Rc cannot cross threads

Rc counts owners on one thread only. The closure borrows it from many threads at once.

let shared = Rc::new(vec![1, 2, 3]);
let total: i32 = (0..3).into_par_iter().map(|i| shared[i]).sum();
// error[E0277]: `Rc<Vec<i32>>` cannot be shared between threads safely

Arc is the fix

Arc counts owners with an atomic counter. That count is safe to share.

let shared = Arc::new(vec![1, 2, 3]);
let total: i32 = (0..3).into_par_iter().map(|i| shared[i]).sum();
println!("total = {}", total);

RefCell cannot cross either

RefCell checks borrows as the program runs. Two threads could pass the check at the same moment.

let counter = RefCell::new(0);
(1..=4).into_par_iter().for_each(|n| {
    *counter.borrow_mut() += n;
});
// error[E0277]: `RefCell<i32>` cannot be shared between threads safely

An atomic needs no lock

Each worker adds with fetch_add. The adds cannot collide.

let evens = AtomicUsize::new(0);
(1..=1000).into_par_iter().for_each(|n| {
    if n % 2 == 0 {
        evens.fetch_add(1, Ordering::Relaxed);
    }
});
println!("evens = {}", evens.load(Ordering::Relaxed));

Tell rayon the smallest chunk

with_min_len sets a floor for splitting. Chunks stay at 100 values or more here.

let total: i32 = (0..1000).into_par_iter().with_min_len(100).sum();
println!("sum = {}", total);

Example

// Cargo.toml: rayon = "1"
use rayon::prelude::*;
use std::collections::BTreeMap;

fn main() {
    let phrases = vec!["rust", "is", "fast"];
    let counts: BTreeMap<char, usize> = phrases
        .par_iter()
        .fold(BTreeMap::new, |mut map, word| {
            for letter in word.chars() {
                *map.entry(letter).or_insert(0) += 1;
            }
            map
        })
        .reduce(BTreeMap::new, |mut a, b| {
            for (letter, n) in b {
                *a.entry(letter).or_insert(0) += n;
            }
            a
        });
    for (letter, n) in &counts {
        println!("{}: {}", letter, n);
    }
}

Key points

  • The rule behind the errors has two halves.
  • Moved values must be Send.
  • Shared values must be Sync.
  • Read the error from its first line.
  • It names the type that failed.
  • It names the rule it broke.
  • Swap Rc for Arc.
  • Swap RefCell for a Mutex or an atomic.
  • The shape of the code barely changes.
  • Splitting is not free.
  • Each task costs scheduling work.
  • Tiny values can run slower in parallel than in sequence.
  • Sharing less beats locking more.
  • Give each worker its own data.
  • Join the results at the end.
  • These errors are the feature.
  • In Rust, a data race fails the build.
  • The error text is long.
  • A rayon compile error stops code that could race, naming the type and the rule it broke.

In this example, three short words go through a parallel chain. fold gives each worker a fresh empty map. Each worker counts the letters in its own words. No map is shared. No lock exists. reduce then merges the small maps into one. Merging adds the counts letter by letter. The BTreeMap prints the letters in alphabetical order. The answer matches a sequential count. The shared state was designed away. The Send & Sync lesson tells the full story of Send and Sync.

Note

It does not ship and crash later.

Try it Yourself

Edit the code, then press Run.

Running is simulated ยท Rust Playground

Exercise

rayon refuses to share an Rc<Vec<i32>> between threads. What is the clean fix?