🎓 A1Learner
🔍

PyTorch Hyperparameter Sweeps

A hyperparameter sweep is a set of training runs that tries planned values of chosen settings. A hyperparameter is a setting chosen before training, not a number the model learns.

The grid is two loops

Three learning rates times two widths plan six runs. The nested loops list every pair exactly once.

results = []
for lr in (0.001, 0.005, 0.02):
    for width in (4, 16):
        results.append((lr, width))
print("Runs planned:", len(results))

Sort by the score

Each result tuple starts with its validation loss. A plain sort then puts the best run first.

results = [(0.3699, 0.02, 4), (0.0004, 0.02, 16), (0.848, 0.001, 16)]
results.sort()
print("Best loss:", results[0][0])
print("Best run: lr", results[0][1], "width", results[0][2])

Loss separates tied accuracy

Two runs from the closing sweep both reach accuracy 1.0. Their losses differ widely, because loss also grades how confident the right answers were.

for loss, acc, lr, width in [(0.2197, 1.0, 0.005, 16), (0.848, 1.0, 0.001, 16)]:
    print(f"lr={lr} width={width} loss={loss} acc={acc}")

Example

import torch
from torch import nn

def make_shapes(per_class, seed):
    g = torch.Generator().manual_seed(seed)
    rows = torch.arange(16).reshape(16, 1)
    cols = torch.arange(16).reshape(1, 16)
    images = []
    labels = []
    for label in range(3):
        for _ in range(per_class):
            img = torch.zeros(16, 16)
            if label == 0:
                top = int(torch.randint(6, 9, (1,), generator=g))
                img[top:top + 3, 2:14] = 1.0
            elif label == 1:
                dist = ((rows - 7.5) ** 2 + (cols - 7.5) ** 2).sqrt()
                img[(dist >= 4.0) & (dist <= 5.5)] = 1.0
            else:
                img[4:12, 4:6] = 1.0
                img[4:12, 10:12] = 1.0
                img[4:6, 4:12] = 1.0
                img[10:12, 4:12] = 1.0
            img = img + 0.05 * torch.rand(16, 16, generator=g)
            images.append(img)
            labels.append(label)
    x = torch.stack(images).unsqueeze(1)
    y = torch.tensor(labels)
    perm = torch.randperm(len(labels), generator=g)
    return x[perm], y[perm]



def run_one(x, y, x_val, y_val, lr, width):
    torch.manual_seed(0)
    model = nn.Sequential(
        nn.Linear(256, width),
        nn.ReLU(),
        nn.Linear(width, 3),
    )
    loss_fn = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=lr)
    for epoch in range(10):
        loss = loss_fn(model(x), y)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    model.eval()
    with torch.no_grad():
        scores = model(x_val)
        val_loss = loss_fn(scores, y_val).item()
        val_acc = (scores.argmax(dim=1) == y_val).float().mean().item()
    return round(val_loss, 4), round(val_acc, 4)


x_train, y_train = make_shapes(30, 100)
x_val, y_val = make_shapes(15, 200)
x_train = x_train.flatten(1)
x_val = x_val.flatten(1)

results = []
for lr in (0.001, 0.005, 0.02):
    for width in (4, 16):
        val_loss, val_acc = run_one(x_train, y_train, x_val, y_val, lr, width)
        results.append((val_loss, val_acc, lr, width))

results.sort()
print("Rank | lr | width | val loss | val accuracy")
for rank, (val_loss, val_acc, lr, width) in enumerate(results, start=1):
    print(f"{rank} | {lr} | {width} | {val_loss} | {val_acc}")

Key points

  • Hyperparameter: a setting chosen before training, like learning rate or layer width
  • Grid: the planned list of setting pairs a sweep will try
  • Use one training function for every run, so only the settings change
  • Rank runs by validation loss, measured on data no run trained on
  • Accuracy can tie at 1.0 while validation loss still separates the runs
  • A sweep replaces one guessed setting with a table of measured ones

In this example, a six-run sweep trains on the shapes data. The training set holds 90 images and the validation set holds 45, and both come from fixed seeds. Each run flattens its images and trains a two-layer network for 10 epochs. run_one resets seed 0 before building its model, so every run starts alike. Only the learning rate and the hidden width change between runs. The ranked table puts learning rate 0.02 with width 16 first, at loss 0.0004. Width 16 beats width 4 at every learning rate in the table. One run reaches accuracy 1.0 yet ranks fifth by loss, at 0.848. Its right answers were hesitant, and the loss noticed what the accuracy missed.

Note

Every run in this sweep starts from seed 0, so the table compares settings, not lucky starts

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

Exercise

In a sweep, what should the ranking score be measured on?