PyTorch Intermediate Intro
PyTorch Intermediate is the band that trains models the way daily practice does. It adds schedules, honest scores, images, sequences, and transfer on top of the Beginner tools.
One fixed rate
A plain optimizer keeps one learning rate for the whole run. The rate never adapts as the model gets close.
torch.manual_seed(1)
model = nn.Linear(3, 1)
optimizer = torch.optim.Adam(model.parameters(), lr=0.05)
print(optimizer.param_groups[0]["lr"])A schedule halves the rate
StepLR multiplies the rate by gamma after every step_size epochs. Twenty epochs at step_size 20 halve 0.05 to 0.025.
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=20, gamma=0.5)
for epoch in range(20):
optimizer.step()
scheduler.step()
print(scheduler.get_last_lr()[0])Print the rate with the losses
A report line can carry the current rate beside the losses. The reader sees the schedule working, not just the losses falling.
lr = 0.025
print(f"{15:>5} | {lr:.4f}")Example
import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset
torch.manual_seed(0)
rows = 160
rooms = torch.randint(1, 6, (rows,)).float()
area = torch.rand(rows) * 2000 + 500
age = torch.rand(rows) * 40
noise = torch.randn(rows) * 8
price = 60 + 18 * rooms + 0.09 * area - 1.2 * age + noise
features = torch.stack([rooms, area, age], dim=1)
targets = price.reshape(-1, 1)
order = torch.randperm(rows)
train_idx, val_idx = order[:128], order[128:]
train_x, train_y = features[train_idx], targets[train_idx]
val_x, val_y = features[val_idx], targets[val_idx]
mean = train_x.mean(dim=0)
std = train_x.std(dim=0)
train_x = (train_x - mean) / std
val_x = (val_x - mean) / std
train_set = TensorDataset(train_x, train_y)
loader = DataLoader(train_set, batch_size=32, shuffle=True)
torch.manual_seed(1)
model = nn.Sequential(
nn.Linear(3, 16),
nn.ReLU(),
nn.Linear(16, 1),
)
loss_fn = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.05)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=20, gamma=0.5)
print("Epoch | LR | Train loss | Val loss")
for epoch in range(1, 61):
model.train()
for batch_x, batch_y in loader:
loss = loss_fn(model(batch_x), batch_y)
optimizer.zero_grad()
loss.backward()
optimizer.step()
scheduler.step()
if epoch % 15 == 0:
model.eval()
with torch.no_grad():
train_loss = loss_fn(model(train_x), train_y).item()
val_loss = loss_fn(model(val_x), val_y).item()
lr = scheduler.get_last_lr()[0]
print(f"{epoch:>5} | {lr:.4f} | {train_loss:>10.1f} | {val_loss:>8.1f}")Key points
- Prerequisite: the PyTorch Beginner course, ending with its house-price capstone
- A learning-rate schedule changes the step size during training instead of fixing it
- Use StepLR to multiply the rate by a factor after a set number of epochs
- Use scheduler.step() once per epoch, after optimizer.step(), to move the rate
- Use scheduler.get_last_lr() to read the rate now in use
- Validation loss: the loss on rows held back from training, watched beside the training loss
- This band adds images, sequences, embeddings, and transfer after the training lessons
In this example, the house data matches the Beginner capstone exactly. One seed builds 160 houses, and the price follows the same fixed rule plus noise. A shuffled split holds back 32 rows for validation. The features are scaled with the training rows' mean and std. The model is the same MLP with one hidden layer of 16. Adam starts at a learning rate of 0.05. A StepLR schedule halves that rate every 20 epochs. Every 15 epochs, the line prints the epoch, the current rate, and both losses. The rate steps down from 0.0500 to 0.0063 across the run. Both losses fall as training proceeds, with the validation loss ending at 336.6. The rest of this course teaches each piece of this discipline in depth.
Note
Every example in this course still runs on CPU with fixed seeds and synthetic data
Try it Yourself
Edit the code, then press Run.
Running is simulated · Python Playground
Exercise
What does a learning-rate schedule do during training?