🎓 A1Learner
🔍

PyTorch Dropout

Dropout is a layer that randomly switches off units during training. No single unit can become load-bearing, because any unit may vanish on the next pass.

One unit doing all the work

A model can lean on a few strong units. Those units fit the training data well and new data poorly.

out = torch.relu(layer(x))
# if one unit holds the key signal, the model breaks when data shifts

Dropout in training mode

The seed is set right before the call, so this exact pattern repeats. Zeros appear, and the survivors print as 2.0 because p is 0.5.

import torch

drop = torch.nn.Dropout(p=0.5)
drop.train()
torch.manual_seed(3)
print(drop(torch.ones(6)).tolist())

Dropout in eval mode

eval() switches dropout off. The same input passes through unchanged, twice in a row.

import torch

drop = torch.nn.Dropout(p=0.5)
drop.eval()
print(drop(torch.ones(6)).tolist())
print(drop(torch.ones(6)).tolist())

Example

import torch

drop = torch.nn.Dropout(p=0.5)
x = torch.ones(8)

drop.train()
torch.manual_seed(1)
print(f"Train call 1: {drop(x).tolist()}")
torch.manual_seed(2)
print(f"Train call 2: {drop(x).tolist()}")

drop.eval()
first = drop(x)
second = drop(x)
print(f"Eval call 1: {first.tolist()}")
print(f"Eval call 2: {second.tolist()}")
print(f"Eval calls equal: {torch.equal(first, second)}")

Key points

  • Unit: one output value of a layer
  • Use torch.nn.Dropout(p=...) to build the layer, where p is the chance of switching a unit off
  • Surviving units are scaled up, so the total stays about the same
  • Dropout is active only in training mode, and train() turns it on
  • eval() turns dropout off, so predictions use every unit
  • Dropout spreads the work across units instead of letting a few units carry the model

In this example, a Dropout layer with p 0.5 receives eight ones. In training mode, each call is seeded right before it runs. The two training calls print different patterns of zeros and twos. The twos appear because surviving units are doubled at p 0.5. In eval mode, both calls print the input unchanged. The last line confirms the two eval calls are equal.

Note

At test time dropout stays off, and the trained model runs with all units on.

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

Exercise

What happens to dropout when a model is switched to eval mode?