🎓 A1Learner
🔍

Rust Deadlock Detection

A deadlock is a circle of waiting between threads. Rust cannot stop a deadlock, so working programmers build witnesses so no important wait goes unwatched.

The smallest deadlock: one thread, one lock, taken twice

A std Mutex is not re-entrant. It does not know one key fact. It does not care about it, either. The holder and the waiter are the same thread:

let m = Mutex::new(1);
let _first = m.lock().unwrap();
let _second = m.lock().unwrap(); // never returns: this thread now waits
                                   // for a guard only it could drop

try_lock turns a freeze into an answer

try_lock never waits: it hands you the guard, or an error saying the lock is held. The match turns a would-be hang into a printed fact:

use std::sync::Mutex;
fn main() {
    let m = Mutex::new(1);
    let _held = m.lock().unwrap();
    let answer = match m.try_lock() {
        Ok(_) => "got the lock",
        Err(_) => "lock is busy: waiting would risk a hang",
    };
    println!("{}", answer);
}

recv_timeout makes silence speak

On channels, the watchdog is a deadline: a reply that does not arrive in time is itself the report:

match rx.recv_timeout(Duration::from_secs(2)) {
    Ok(value) => println!("worker answered: {}", value),
    Err(_) => println!("no answer in 2s: treat the worker as stuck"),
}

Example

use std::sync::mpsc;
use std::sync::Mutex;
use std::thread;
use std::time::Duration;

fn main() {
    let (tx, rx) = mpsc::channel::<&str>();
    let stuck = thread::spawn(move || {
        let m = Mutex::new(0);
        let _first = m.lock().unwrap();
        let _second = m.lock().unwrap();
        tx.send("finished").unwrap();
    });
    match rx.recv_timeout(Duration::from_millis(300)) {
        Ok(msg) => println!("worker says: {}", msg),
        Err(_) => println!("watchdog: no word in 300ms — the worker is stuck"),
    }
    println!("main carries on; worker still unfinished = {}", !stuck.is_finished());
}

Key points

  • Name the failure.
  • A deadlock is a circle of waiting.
  • Every thread in the circle holds something.
  • The next thread needs that thing.
  • No thread outside the circle can break in
  • A deadlock needs four things together.
  • Threads hold locks while they wait.
  • A lock cannot be taken by force.
  • Locks are not shared.
  • The waiting forms a circle.
  • Remove any one of the four and deadlock is impossible — the shared lock order from Dining Philosophers removes the circle
  • The detection mindset is simple.
  • Every long wait should have a witness.
  • A timeout can be that witness.
  • A heartbeat counter can be one, too.
  • Keep it in an atomic.
  • Other threads can watch it.
  • Or use a watchdog thread.
  • Its only job is to notice silence.
  • It reports that silence
  • When a program hangs, check the boring causes first.
  • A sender may never have been dropped.
  • That keeps recv loops alive forever.
  • A guard may be held across a channel send.
  • It may be held across a recv.
  • That starves the other side.
  • Two locks may be taken in opposite orders.
  • That is the classic circle
  • A hung program is the deadlock smell.
  • Its CPUs are idle.
  • It shows no fresh output.
  • List the facts before changing code.
  • List who holds what.
  • List who waits for what.
  • The circle is usually visible on paper first
  • A deadlocked thread cannot even panic its way out, and ending the process is usually the only cure.

In this example, the worker thread deadlocks itself on purpose: it locks a Mutex, then locks the same Mutex again. A std Mutex is not re-entrant. So the second call waits for a guard. Only this same thread could ever drop it. The message "finished" is never sent. The watchdog is main itself, waiting with recv_timeout: after 300 milliseconds it prints its report instead of hanging forever. The last line asks the worker's handle whether it finished — false, and it always will be. main then simply ends, and when the main thread exits, the process exits, stuck thread and all. A real program would log the same report. Then it would fix the cause. Use one lock, taken once. Or use a lock order. Every thread must share that order.

Note

That is why the witnesses — a few lines each — are worth building before you need them.

Try it Yourself

Edit the code, then press Run.

Running is simulated · Rust Playground

Exercise

Your multithreaded program has frozen: CPU idle, nothing printing. Which first step fits the detection mindset?