🎓 A1Learner
🔍

PyTorch Label Smoothing

Label smoothing is a setting that softens the target inside cross-entropy loss. The correct class gets most of the target, and every class keeps a small share.

A hard target, softened

The middle class is the correct one here. Smoothing takes a small share from it and spreads the share over all classes.

eps = 0.1
classes = 3
hard = [0.0, 1.0, 0.0]
soft = [(1 - eps) * v + eps / classes for v in hard]
print([round(v, 4) for v in soft])

Turning smoothing on

One argument turns smoothing on. The rest of the training loop stays the same.

loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=0.1)

The cost of full confidence

This prediction is correct and confident. The smoothed loss is still much larger, because the target itself is no longer 1.0.

import torch

logits = torch.tensor([[4.0, 0.0, 0.0]])
target = torch.tensor([0])
plain = torch.nn.CrossEntropyLoss()(logits, target)
smooth = torch.nn.CrossEntropyLoss(label_smoothing=0.1)(logits, target)
print(f"Plain: {plain.item():.4f}")
print(f"Smoothed: {smooth.item():.4f}")

Example

import torch

torch.manual_seed(0)
centers = torch.tensor([[2.0, 0.0], [-1.0, 2.0], [-1.0, -2.0]])
x = torch.cat([c + torch.randn(30, 2) * 0.6 for c in centers])
y = torch.cat([torch.full((30,), i) for i in range(3)])


def train(smoothing):
    torch.manual_seed(1)
    model = torch.nn.Linear(2, 3)
    optimizer = torch.optim.Adam(model.parameters(), lr=0.05)
    loss_fn = torch.nn.CrossEntropyLoss(label_smoothing=smoothing)
    for step in range(150):
        optimizer.zero_grad()
        loss = loss_fn(model(x), y)
        loss.backward()
        optimizer.step()
    with torch.no_grad():
        probs = torch.softmax(model(x), dim=1)
        confidence = probs.max(dim=1).values.mean().item()
    return loss.item(), confidence


for smoothing in [0.0, 0.1]:
    loss, confidence = train(smoothing)
    print(f"Smoothing {smoothing}: final loss {loss:.4f}, "
          f"mean confidence {confidence:.4f}")

Key points

  • Target: the answer the loss trains the model toward, written as a spread over classes
  • Use CrossEntropyLoss(label_smoothing=...) to turn smoothing on
  • Confidence: the largest class probability a model gives one example, read with max()
  • A hard target gives the correct class a full 1.0 and every other class 0.0
  • Smoothing 0.1 with 3 classes moves the correct target to 0.9333, as the first snippet shows
  • A smoothed target stops rewarding endless confidence, so the model stays a little unsure

In this example, 90 rows form three clusters and the targets name the clusters. A train function builds the same Linear model for each smoothing value. Smoothing 0 ends with loss 0.0110 and a mean confidence of 0.9894. Smoothing 0.1 ends with loss 0.3280 and a mean confidence of 0.9087. The higher smoothed loss is expected, because its target is softer. The confidence drop is the real effect. The smoothed model stops just short of certainty.

Note

Smoothed and plain losses use different targets, so their sizes are not a fair race.

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

Exercise

What does label smoothing change?