🎓 A1Learner
🔍

PyTorch Seeding Everything

Seeding is setting every random stream in a run to a known starting point. A fully seeded run repeats exactly, because no stream is left to chance.

Two draws, no seed

Random draws with no seed cannot repeat. Two calls in a row almost never match.

first = torch.randn(3)
second = torch.randn(3)
print("Two draws equal:", torch.equal(first, second))

One seed repeats a draw

Setting the same seed again restarts the same stream. The second draw matches the first, value for value.

torch.manual_seed(42)
first = torch.randn(3)
torch.manual_seed(42)
second = torch.randn(3)
print("First:", [round(v, 2) for v in first.tolist()])
print("Second:", [round(v, 2) for v in second.tolist()])
print("Equal:", torch.equal(first, second))

The loader gets its own generator

A DataLoader shuffles from its generator, not from the global stream. The generator needs its own seed.

g = torch.Generator().manual_seed(7)
loader = DataLoader(data, batch_size=8, shuffle=True,
                    generator=g)

Workers need seeds too

Each worker is a separate process with its own streams. worker_init_fn runs at worker start-up, so it can seed the worker there.

def seed_worker(worker_id):
    torch.manual_seed(7 + worker_id)

loader = DataLoader(data, batch_size=8, num_workers=2,
                    worker_init_fn=seed_worker)

Example

import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset


def run_once():
    torch.manual_seed(7)
    x = torch.randn(24, 3)
    y = x @ torch.tensor([1.0, -2.0, 0.5]) + 0.1

    torch.manual_seed(7)
    model = nn.Linear(3, 1)
    generator = torch.Generator().manual_seed(7)
    loader = DataLoader(
        TensorDataset(x, y.reshape(-1, 1)),
        batch_size=8, shuffle=True, generator=generator,
    )
    loss_fn = nn.MSELoss()
    optimizer = torch.optim.SGD(model.parameters(), lr=0.05)
    losses = []
    for xb, yb in loader:
        loss = loss_fn(model(xb), yb)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        losses.append(round(loss.item(), 4))
    return losses


first = run_once()
second = run_once()
print("Run 1 losses:", first)
print("Run 2 losses:", second)
print("Identical:", first == second)

Key points

  • Seed: a starting number that fixes the order of one random stream
  • Use torch.manual_seed(n) to seed the global torch stream
  • Use a torch.Generator with its own manual_seed to seed one loader
  • DataLoader shuffling draws its order from the loader's generator
  • Seeding every stream is what makes two runs print the same losses
  • Worker seeds: each loader worker needs a seed once num_workers is above zero

In this example, run_once is one full training run, and the program calls it twice. Seed 7 is set before the data is drawn, so both calls build the same 24 rows. Seed 7 is set again before the model is built, so both models start from the same numbers. A separate generator with seed 7 drives the DataLoader shuffle. The loader serves three batches of eight rows, and each batch loss is rounded and saved. Both runs print the same three losses, and the equality check prints True. One seed would not have been enough, because data, model, and shuffle draw from different streams.

Note

With num_workers at its default of zero, no worker seeds are needed

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

Exercise

A script seeds torch but not its DataLoader. What can still change between runs?