🎓 A1Learner
🔍

Rust TT Munchers

A TT muncher is a recursive macro that eats its input one token at a time. Each step matches a few tokens, does one small job, and calls itself on the tokens left over.

Eat one token, keep the rest

The whole technique is this one arm shape — work on the first token, recurse on the tail:

($first:tt $($rest:tt)*) => {
    // ... do one job with $first ...
    spell!($($rest)*); // then munch what is left
};

The empty arm stops the meal

With no tokens left, this arm matches and the recursion ends:

() => {};

Each step sees a smaller input

Watch the call shrink. This trace is a comment, not a program:

// spell!(R u s t)
// -> print R, then spell!(u s t)
// -> print u, then spell!(s t)
// -> print s, then spell!(t)
// -> print t, then spell!()  -- empty arm, stop

Bundle arguments a step does not read

Group spare arguments into one bundle. A step can pass the bundle on whole, as a single tt:

($bundle:tt, $($rest:tt)*) => {
    finish!($bundle, $($rest)*)
};

Example

macro_rules! spell {
    () => {};
    ($first:tt $($rest:tt)*) => {
        println!("letter: {}", stringify!($first));
        spell!($($rest)*);
    };
}
fn main() {
    spell!(R u s t);
}

Key points

  • TT muncher: a recursive macro that works from the front of its input
  • It matches a few tokens and does one small job
  • Then it calls itself on what is left
  • The leftover input is always captured as $($rest:tt)*
  • A tt repetition is the only capture that keeps tokens exactly as written, unparsed
  • An empty arm, () => {}, is the stop rule: when nothing is left, the munching ends
  • Munching is slow for long inputs
  • Every step re-matches the whole remaining input
  • The cost grows quadratically
  • Keep munched inputs short
  • Prefer plain repetition $( ... )* when every piece has the same shape
  • Reach for a muncher only when pieces differ or must be read in order
  • TT bundling: wrap arguments that middle steps do not need into one ( ) group
  • A step can then forward them as a single $bundle:tt
  • It never reads them
  • A TT muncher is the strongest parsing trick macro_rules! has.

In this example, spell munches its input one token at a time. The call spell!(R u s t) matches the second arm: $first takes R, and $($rest:tt)* keeps u s t exactly as written. The arm prints letter: R with stringify!, then calls spell! on the rest. The same thing happens for u and for s. When the call is spell!(t), the rest is empty. The next call, spell!(), matches the first arm and stops. Four tokens, four munching steps, four printed lines. The Rust Macro Recursion lesson shows how to raise the recursion limit.

Note

Munchers and recursion share one budget

Each munch is one level of macro recursion

The default limit is 128 levels

Try it Yourself

Edit the code, then press Run.

Running is simulated · Rust Playground

Exercise

Why is the leftover input captured as $($rest:tt)* instead of a list of expressions?