🎓 A1Learner
🔍

Rust Tokio Framing

Framing is the step that turns a stream of byte chunks into whole messages. TCP reads return chunks of any size.

The problem: chunks, not messages

One read can return any amount of data — half a frame, or several frames at once.

let n = stream.read(&mut buf).await?;
// n bytes arrived: maybe half a frame,
// maybe three frames stuck together.

A frame is just data

The tutorial's Frame type for the Redis protocol holds data only. Its meaning comes one level up.

enum Frame {
    Simple(String),
    Error(String),
    Integer(u64),
    Bulk(Bytes),
    Null,
    Array(Vec<Frame>),
}

Keep a buffer on the connection

pub struct Connection {
    stream: BufWriter<TcpStream>,
    buffer: BytesMut,
}

Try to parse, then read more

read_frame loops. It parses what you have. If the frame is incomplete, it reads more bytes and tries again.

if let Some(frame) = self.parse_frame() {
    return Ok(Some(frame));
}
// Not a full frame yet — read more into the buffer.
if 0 == self.stream.read_buf(&mut self.buffer).await? {
    // 0 means the peer closed the connection.
}

Cut the frame out of the buffer

A check pass finds the frame's end. advance then throws those bytes away. It keeps the rest for the next frame.

let len = buf.position() as usize; // end of the frame
let frame = Frame::parse(&mut buf)?;
self.buffer.advance(len); // discard the frame's bytes

Writing a frame, piece by piece

A bulk frame on the wire is a $ head, the length, then the data. Buffer the pieces, flush once.

self.stream.write_u8(b'$').await?;
self.stream.write_all(len.as_bytes()).await?;
self.stream.write_all(&data).await?;
self.stream.flush().await?;

Example

// Cargo.toml: bytes = "1" (framing needs no socket — just the buffer)
use bytes::BytesMut;

fn encode(buffer: &mut BytesMut, frame: &str) {
    buffer.extend_from_slice(frame.as_bytes());
    buffer.extend_from_slice(b"\n");
}

fn next_frame(buffer: &mut BytesMut) -> Option<String> {
    let newline = buffer.iter().position(|b| *b == b'\n')?;
    let mut line = buffer.split_to(newline + 1);
    line.truncate(newline);
    Some(String::from_utf8(line.to_vec()).unwrap())
}

fn main() {
    let mut buffer = BytesMut::new();
    encode(&mut buffer, "hello");
    encode(&mut buffer, "tokio framing");
    while let Some(frame) = next_frame(&mut buffer) {
        println!("frame: {}", frame);
    }
    println!("buffer bytes left = {}", buffer.len());
}

Key points

  • Framing: turning a stream of bytes into a stream of frames — one frame is one whole message
  • A frame carries data only, no meaning.
  • The Frame enum is just Simple, Error, Integer, Bulk, Null, or Array.
  • Commands are parsed one level up.
  • read_buf reads straight into a BytesMut.
  • It moves the cursor for you.
  • There is no hand-kept position.
  • And there is no zero-filling of fresh capacity.
  • A read returning 0 means the peer closed the connection.
  • An empty buffer is a clean end.
  • A non-empty buffer means a frame was cut in half.
  • That is an error.
  • Writes are buffered too.
  • A BufWriter collects the small pieces of a frame and flushes once.
  • It does not make one system call per piece.
  • For ready-made framing, the tokio-util crate offers Framed.
  • It also offers a Decoder/Encoder pair — a codec.
  • So you describe the frame shape once.
  • You do not hand-write the loop.
  • TCP has no message boundaries at all.

In this example, encode writes two text frames into one BytesMut buffer. Each frame ends with a newline. That is the simplest frame shape there is: a line. next_frame looks for the next newline in the buffer. If there is none, it returns None. That is the "not a full frame yet" answer. A read_frame would get it before reading more. When a newline is found, split_to cuts that line off the front of the buffer. The newline is included in the cut. And truncate drops the newline itself. The loop prints both frames: hello, then tokio framing. The buffer ends empty. Every byte was consumed by exactly one frame. On a real socket, the same cut-and-keep-the-rest logic sits inside read_frame, with reads refilling the buffer.

Note

Any "messages" in a TCP program exist because the framing code created them.

Try it Yourself

Edit the code, then press Run.

Running is simulated · Rust Playground

Exercise

One read from a socket returns 100 bytes. What can you say about the frames inside?