Rust Token Streams in Depth

A TokenStream is the stream of token pieces that a procedural macro takes in and hands back. A second TokenStream type lives in the proc-macro2 crate.

Four kinds of token trees

Ident is a name or a keyword. Punct is one mark, like + or #. Literal is a value, like 42 or "hi". Group is brackets with more trees inside:

pub enum TokenTree {
    Ident(Ident),
    Punct(Punct),
    Literal(Literal),
    Group(Group),
}

A macro that counts trees

This function-like macro counts the trees at the top level. It converts the stream first. Then it counts the pieces:

#[proc_macro]
pub fn count_tokens(input: TokenStream) -> TokenStream {
    let stream = proc_macro2::TokenStream::from(input);
    let n = stream.into_iter().count();
    quote!(#n).into()
}

Words count one by one

Three idents go in. The macro returns the number 3. The program prints it:

println!("three words: {}", count_tokens!(one two three));

Brackets hide their insides

The group counts once. Its two words stay hidden from the count:

println!("a group is one tree: {}", count_tokens!(one (two three) four));

Each mark is its own tree

A plus sign is a Punct tree. Two plus signs add two to the count:

println!("plus is its own tree: {}", count_tokens!(a + b + c));

See the generated code

Reading raw tokens gets old fast. The Debugging Macros lesson shows cargo expand. It prints your crate after every macro has run:

// in the user crate:
// $ cargo expand

Example

// toolkit holds count_tokens.
// Built and run with quote 1.0.47 and proc-macro2 1.0.107.
use toolkit::count_tokens;

fn main() {
    println!("three words: {}", count_tokens!(one two three));
    println!("a group is one tree: {}", count_tokens!(one (two three) four));
    println!("plus is its own tree: {}", count_tokens!(a + b + c));
}

Key points

A token tree is one piece of a stream
It is an ident, a punctuation mark, a literal, or a whole group
A group is a pair of brackets with a stream inside
The group counts as one tree
It does not matter how much it holds
proc_macro::TokenStream works only inside a macro
proc_macro2::TokenStream works anywhere
Helpers and unit tests can use it
The two stream types convert with From and Into
Macro code converts at the border
It works with proc_macro2 inside
parse_macro_input! parses the input for you
On bad input it returns a compile error
It stops the macro early

In this example, count_tokens! is a function-like macro. It receives the tokens between its brackets as one stream. It converts the stream to proc_macro2. Then it counts the trees at the top level. The first call holds three idents. It prints 3. The second call holds an ident, a group, and an ident. The group counts once. It also prints 3. The third call holds three idents and two plus marks. Each plus is a Punct tree. It prints 5. The macro returns the count as a literal. The finished program only sees a number.

Note

Every token carries a span

The span is its source position

Tokens from quote! point at the macro call by default

So generated names can clash with user names

Real crates choose spans with care

Try it Yourself

Edit the code, then press Run.

Running is simulated · Rust Playground

// toolkit holds count_tokens.
// Built and run with quote 1.0.47 and proc-macro2 1.0.107.
use toolkit::count_tokens;

fn main() {
    println!("three words: {}", count_tokens!(one two three));
    println!("a group is one tree: {}", count_tokens!(one (two three) four));
    println!("plus is its own tree: {}", count_tokens!(a + b + c));
}

Exercise

What does count_tokens!(a (b c d) e) return?

6 5 3 1