🎓 A1Learner
🔍

PyTorch A Model by Hand

A model by hand is a fit built from raw tensors and explicit update steps. It trains w and b for the line y = wx + b without nn or an optimizer.

One step, all by hand

The prediction starts at 0 and the target is 5. Both gradients come out as -10.

w = torch.tensor(0.0, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)
x = torch.tensor([1.0])
y = torch.tensor([5.0])
loss = ((w * x + b - y) ** 2).mean()
loss.backward()
print(w.grad)
print(b.grad)

Loss at the start

The loss squares each miss and averages the squares. A small loss means close predictions.

x = torch.tensor([0.0, 0.5, 1.0])
y = 3 * x + 2
w = torch.tensor(0.0, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)
predictions = w * x + b
loss = ((predictions - y) ** 2).mean()
print(f"{loss.item():.2f}")

Clearing a gradient

A gradient stays in .grad until it is cleared. Clearing sets it back to zero for the next step.

w = torch.tensor(1.0, requires_grad=True)
(w * 2).backward()
print(w.grad)
w.grad.zero_()
print(w.grad)

Example

import torch

torch.manual_seed(0)

# True rule behind the data: y = 3x + 2, plus small noise.
x = torch.linspace(0, 1, 20)
y = 3 * x + 2 + torch.randn(20) * 0.05

w = torch.tensor(0.0, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)
learning_rate = 0.5

for step in range(200):
    predictions = w * x + b
    loss = ((predictions - y) ** 2).mean()
    loss.backward()
    with torch.no_grad():
        w -= learning_rate * w.grad
        b -= learning_rate * b.grad
    w.grad.zero_()
    b.grad.zero_()
    if step == 0:
        print(f"First loss: {loss.item():.4f}")

print(f"Final loss: {loss.item():.4f}")
print(f"Learned w: {w.item():.2f}")
print(f"Learned b: {b.item():.2f}")
print("True w: 3.00")
print("True b: 2.00")

Key points

  • Parameter: a number in the model that training adjusts, like w and b
  • Loss: one number that scores how wrong the predictions are
  • Use loss.backward() to fill .grad for each parameter
  • Learning rate: the step size that scales each update
  • Use w.grad.zero_() to clear a gradient before the next step
  • Training by hand: predict, score, get gradients, update, and repeat

In this example, the seed is set first, so the noise is the same on every run. The data follows y = 3x + 2, plus small random noise. w and b start at 0 with requires_grad=True. Each of the 200 steps builds predictions and a mean squared loss. backward() fills the two gradients. The updates then move w and b against their gradients, scaled by 0.5. zero_() clears both gradients for the next step. The first loss prints inside the loop at step 0. The final loss is far smaller than the first. The learned w and b print rounded to two decimals. They land close to the true 3 and 2, not exactly on them. Every step here is written out by hand. Later lessons hand this repeated work to nn modules and optimizers.

Note

The updates run under no_grad, so autograd does not track the fixes

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

Exercise

In the by-hand loop, what does loss.backward() do?