PyTorch Advanced Intro
PyTorch Advanced is the band that opens the machinery under training. Its lessons write gradients by hand, build attention from tensors, export models, and generate data.
The sine LSTM from Intermediate
An LSTM is a recurrent network with gates that protect its memory. This pass reads three sine windows and prints the output shapes.
import torch
from torch import nn
torch.manual_seed(0)
lstm = nn.LSTM(1, 16, batch_first=True)
batch = torch.randn(3, 12, 1)
output, _ = lstm(batch)
print("Output shape:", tuple(output.shape))
print("Last step shape:", tuple(output[:, -1].shape))One gradient from autograd
tanh squashes a value into the range from minus one to one. Autograd already knows the gradient rule for tanh, so backward() fills x.grad with no hand-written code.
x = torch.tensor([0.0, 1.0], requires_grad=True)
y = torch.tanh(x)
print("Forward:", [round(v, 3) for v in y.tolist()])
y.sum().backward()
print("Gradient:", [round(v, 3) for v in x.grad.tolist()])The tanh inside the model
The model applies tanh to the last LSTM step. The linear head then turns that row into one prediction.
class SinePredictor(nn.Module):
def __init__(self, hidden):
super().__init__()
self.lstm = nn.LSTM(1, hidden, batch_first=True)
self.head = nn.Linear(hidden, 1)
def forward(self, x):
output, _ = self.lstm(x)
squashed = torch.tanh(output[:, -1])
return self.head(squashed)Example
import torch
from torch import nn
def make_windows(wave, width):
inputs = []
targets = []
for start in range(len(wave) - width):
inputs.append(wave[start:start + width])
targets.append(wave[start + width])
return torch.stack(inputs).unsqueeze(-1), torch.stack(targets).unsqueeze(-1)
class SinePredictor(nn.Module):
def __init__(self, hidden):
super().__init__()
self.lstm = nn.LSTM(1, hidden, batch_first=True)
self.head = nn.Linear(hidden, 1)
def forward(self, x):
output, _ = self.lstm(x)
squashed = torch.tanh(output[:, -1])
return self.head(squashed)
wave = torch.sin(torch.arange(400) * 0.1)
x, y = make_windows(wave, 12)
x_train, y_train = x[:280], y[:280]
x_val, y_val = x[280:], y[280:]
torch.manual_seed(0)
model = SinePredictor(16)
loss_fn = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
for epoch in range(50):
loss = loss_fn(model(x_train), y_train)
optimizer.zero_grad()
loss.backward()
optimizer.step()
with torch.no_grad():
val_error = loss_fn(model(x_val), y_val).item()
print("Train windows:", len(x_train))
print("Validation windows:", len(x_val))
print("Validation MSE:", round(val_error, 4))Key points
- Custom autograd function: a class that supplies its own forward rule and its own backward rule
- Use a custom autograd function when the built-in gradient rule is not the rule the model needs
- Attention: a way for each step of a sequence to read every earlier step
- Export: saving a model in a form that runs outside the training script
- Generation: using a trained model to produce new data step by step
- In short, this band writes the rules that earlier bands took from the library
In this example, the wave holds 400 sine values, cut into windows of width 12. The first 280 windows train the model and the last 108 validate it. SinePredictor wraps an LSTM with hidden size 16 and a linear head. The forward pass applies tanh to the last step before the head, and autograd supplies the gradient for that tanh. Training runs 50 epochs with Adam at rate 0.01. The validation MSE prints as 0.0023. The plain LSTM in PyTorch Intermediate printed 0.002 on the same split. The next lesson writes a gradient like this one by hand, as a custom autograd function with its own forward and backward rules.
Note
This course builds on PyTorch Intermediate and its sine LSTM project
Try it Yourself
Edit the code, then press Run.
Running is simulated · Python Playground
Exercise
In the snippet, why is the printed gradient at x = 1.0 equal to 0.42?