Home / PyTorch Advanced
PyTorch Advanced
Worked by hand: custom autograd and gradcheck, attention and transformers, distributions, VAEs, GANs and diffusion, export and quantization, FX graphs, distributed honestly and distillation — ending in an export capstone. Builds on PyTorch Intermediate.
Start learning ❯❯71 lessons · 17 topics · 0/71 completed
Course contents
Work top to bottom — the sidebar is the course. Topics later in the list assume the ones before them.
PyTorch Advanced Intro
Advanced · ⏱ 8 min · standalone lesson
Autograd in Depth
- PyTorch Custom Autograd Functions
- PyTorch A Custom Gradient, Checked
- PyTorch Gradcheck
- PyTorch Second Derivatives
- PyTorch Tensor Hooks
- PyTorch Jacobians with torch.func
- PyTorch Per-Sample Gradients with vmap
Attention from Scratch
- PyTorch Why Attention
- PyTorch Einsum
- PyTorch Scaled Dot-Product Attention
- PyTorch Attention Masks
- PyTorch nn.MultiheadAttention
- PyTorch Reading Attention Weights
- PyTorch Attention Variants
Transformer Blocks
- PyTorch Positional Encoding
- PyTorch The Encoder Layer
- PyTorch The Decoder Layer
- PyTorch Cross-Attention in Depth
- PyTorch Building a Transformer Classifier
- PyTorch Transformer Gotchas
- PyTorch Encoder–Decoder Project
Custom Modules
Start: PyTorch Writing a Custom Layer ❯Distributions and Sampling
- PyTorch torch.distributions
- PyTorch Reparameterization
- PyTorch Entropy and KL Divergence
- PyTorch Sampling from Logits
Generative Models — VAE
- PyTorch Autoencoder Basics
- PyTorch The Variational Step
- PyTorch Training a VAE
- PyTorch A Latent Space Walk
Generative Models — GAN and Diffusion
- PyTorch The GAN Idea
- PyTorch Training a Tiny GAN
- PyTorch GAN Failure Modes
- PyTorch The Diffusion Idea
- PyTorch A Tiny Diffusion Model
Optimizer Maths
Start: PyTorch What Adam Remembers ❯Export and Deployment
- PyTorch TorchScript Tracing
- PyTorch TorchScript Scripting
- PyTorch Saving Scripted Models
- PyTorch torch.export
- PyTorch Dynamic Quantization
- PyTorch Quantization-Aware Training
- PyTorch Pruning
Performance on CPU
- PyTorch Benchmarking Honestly
- PyTorch Vectorization over Loops
- PyTorch torch.compile on CPU
- PyTorch Channels Last
- PyTorch Profiling with torch.profiler
- PyTorch Precision and Loss Scaling
Graph Mode with FX
Start: PyTorch FX Symbolic Tracing ❯Distributed, Honestly
- PyTorch Why Distributed
- PyTorch All-Reduce by Hand
- PyTorch The DDP Pattern
- PyTorch Model Parallel and Sharding
Knowledge Distillation
- PyTorch The Teacher–Student Idea
- PyTorch Distilling on Synthetic Data
- PyTorch Distillation Results, Honestly
Robustness
Start: PyTorch Adversarial Examples (FGSM) ❯Advanced Training Patterns
Start: PyTorch EMA of Weights ❯PyTorch Advanced Capstone
Advanced · ⏱ 12 min · standalone lesson