🎓 A1Learner
🔍

PyTorch The Validation Loop

A validation loop scores the held-back rows after each epoch. It runs under no_grad in eval mode, so it changes nothing in the model.

Score without training

Eval mode and no_grad make the scoring pass safe. The pass reads the model and changes nothing.

model.eval()
with torch.no_grad():
    val_loss = loss_fn(model(x_val), y_val)
print(f"Val loss: {val_loss.item():.4f}")

Back to train mode

Train mode switches the model back for the next epoch. The training flag reads True in train mode.

model.train()
print(model.training)

The rhythm of one epoch

One epoch trains first and scores second. The scoring pass sits after the update, inside the same epoch loop.

model.train()
# train on the training rows
model.eval()
with torch.no_grad():
    val_loss = loss_fn(model(x_val), y_val)

Example

import torch
import torch.nn as nn

torch.manual_seed(0)
g_train = torch.Generator().manual_seed(1)
g_val = torch.Generator().manual_seed(2)

x_train = torch.linspace(0, 1, 40).reshape(-1, 1)
y_train = 2.0 * x_train + 1.0 + torch.randn(40, 1, generator=g_train) * 0.1
x_val = torch.linspace(0.05, 0.95, 10).reshape(-1, 1)
y_val = 2.0 * x_val + 1.0 + torch.randn(10, 1, generator=g_val) * 0.1

model = nn.Linear(1, 1)
loss_fn = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)

print("Epoch  Train   Val")
for epoch in range(1, 9):
    model.train()
    optimizer.zero_grad()
    loss = loss_fn(model(x_train), y_train)
    loss.backward()
    optimizer.step()
    model.eval()
    with torch.no_grad():
        val_loss = loss_fn(model(x_val), y_val)
    print(f"{epoch}      {loss.item():.4f}  {val_loss.item():.4f}")

Key points

  • Use model.eval() before scoring the held-back rows
  • Use torch.no_grad() so scoring builds no gradients
  • Validation loss: the loss on rows the training never touches
  • Use model.train() to switch back before the next epoch
  • Two losses per epoch show learning and honest scoring side by side

In this example, the training rows and the validation rows follow the same rule. Each epoch trains on the training rows first. The model then switches to eval mode and scores the validation rows under no_grad. The table prints both losses for each epoch. Both losses fall, so the model improves on rows it trains on and on rows it never touches.

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

Exercise

Why does the validation pass run under no_grad?