🎓 A1Learner
🔍

Rust Reading Big Files

Reading a big file in pieces keeps only a small part of it in memory at a time. read_to_string reads a whole file at once.

The whole file at once

read_to_string is the simple tool. It loads everything before your code sees any of it. For the small log in this lesson:

use std::fs;
let text = fs::read_to_string("big.log").expect("read failed");
println!("chars = {}", text.len());

Stream it line by line

BufReader plus lines() walks the same file without loading it. Only one line lives in memory at a time:

use std::fs::File;
use std::io::{BufRead, BufReader};
let reader = BufReader::new(File::open("big.log")?);
for line in reader.lines() {
    let line = line?;
    if line.contains("cat") {
        println!("{line}");
    }
}

Reuse one buffer

read_line appends the next line to a String you already own. Clear it before each pass. No new memory is made per line:

let mut buffer = String::new();
loop {
    buffer.clear();
    let n = reader.read_line(&mut buffer)?;
    if n == 0 { break; }
    println!("got {} bytes: {}", n, buffer.trim_end());
}

Count bytes without text

bytes() hands you one byte at a time, each as a small Result. Counting them never builds a String:

use std::io::Read;
let reader = BufReader::new(File::open("big.log")?);
println!("bytes = {}", reader.bytes().count());

Errors can arrive mid-file

Each line is its own Result. A file with damaged bytes can fail halfway through. It does not fail only at the open:

for line in reader.lines() {
    match line {
        Ok(text) => println!("{text}"),
        Err(e) => println!("failed: {:?}", e.kind()),
    }
}

Example

use std::fs::{self, File};
use std::io::{BufRead, BufReader};

fn main() {
    fs::write("server.log", "INFO start\nWARN disk slow\nINFO ok\nWARN retry\nINFO done\n")
        .expect("could not write the log");
    let file = File::open("server.log").expect("could not open the log");
    let reader = BufReader::new(file);
    let mut warnings = 0;
    let mut total = 0;
    for line in reader.lines() {
        let line = line.expect("could not read a line");
        total += 1;
        if line.contains("WARN") {
            warnings += 1;
            println!("{line}");
        }
    }
    println!("{warnings} warnings in {total} lines");
    fs::remove_file("server.log").expect("could not clean up");
}

Key points

  • Use fs::read_to_string when the file is small: it is one call and the whole text is yours
  • Use a BufReader when the file is big: it keeps only a small buffer in memory
  • Use reader.lines() to walk a big file one line at a time: each line arrives as its own Result
  • Use read_line with one reused String when speed matters most: the same buffer serves every line
  • A read_line result of 0 means the file is finished: that is the signal to stop the loop

In this example, the program writes a five-line log first. It then opens the log through a BufReader. The whole file never sits in memory. Each line arrives on its own. Lines with WARN print and raise the warnings count. Every line raises the total count. The last line reports 2 warnings in 5 lines. A real server log could hold a billion lines. The memory use would stay the same.

Note

BufReader reads ahead in chunks of 8 KB by default.

BufReader::with_capacity makes the chunk bigger for very large files.

Try it Yourself

Edit the code, then press Run.

Running is simulated · Rust Playground

Exercise

Your tool must search a 4 GB log file. Which approach fits?