🎓 A1Learner
🔍

PyTorch Why Convolution

Convolution scans an image with one small pattern detector that slides across every position. A fully connected layer instead gives every output cell its own weight for every pixel.

The MLP bill

A 16 by 16 image flattens to 256 numbers. One Linear layer to 64 cells pays a weight for each input and output pair.

mlp = nn.Linear(256, 64)
print(sum(p.numel() for p in mlp.parameters()))

A shifted bar

The same bar in a new row changes the flat input in many places. An MLP sees a different input, not a moved pattern.

top = torch.zeros(4, 4)
top[0, :] = 1
bottom = torch.zeros(4, 4)
bottom[3, :] = 1
print(int((top.reshape(-1) != bottom.reshape(-1)).sum()))

One kernel everywhere

A convolution layer holds a few kernels, not a weight per pixel. Each 3 by 3 kernel carries nine weights and one bias, so the count is small enough to do by hand.

per_kernel = 3 * 3 + 1
print(8 * per_kernel)

Example

import torch
from torch import nn

torch.manual_seed(0)

image = torch.rand(1, 1, 16, 16)

mlp = nn.Linear(16 * 16, 64)

mlp_count = sum(p.numel() for p in mlp.parameters())
conv_count = 8 * (3 * 3 + 1)
out_side = 16 - 3 + 1

flat = image.reshape(1, 16 * 16)
print("Image shape:", tuple(image.shape))
print("MLP input size:", flat.shape[1])
print("MLP parameters:", mlp_count)
print("Conv parameters:", conv_count)
print("Conv output shape:", (1, 8, out_side, out_side))
print("MLP has more:", mlp_count > conv_count)

Key points

  • Kernel: a small grid of weights that detects one pattern, like an edge
  • Feature map: the grid of scores a kernel leaves as it slides
  • Use convolution to look for the same pattern anywhere in an image
  • Weight sharing: one kernel serves every position, so its cost stays small
  • A shifted pattern is a new pattern to an MLP, but the same pattern to a kernel

In this example, the seed is fixed and a random 1 by 16 by 16 image is built. An MLP layer here is a Linear layer from 256 inputs to 64 cells. Its real parameter count prints as 16448. The conv count is done by hand: each of the eight 3 by 3 kernels carries nine weights and one bias, so the layer holds 80 parameters. A kernel only fits where it fully overlaps the image, so each output map runs 16 minus 3 plus 1, which is 14 cells per side. One small set of shared weights replaces a weight for every pixel pair.

Note

PyTorch convolution does not flip the kernel, so it matches the hand sum in the next lesson

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

Exercise

Why does a conv layer need far fewer parameters than an MLP layer on the same image?