🎓 A1Learner
🔍

PyTorch The Teacher–Student Idea

Knowledge distillation is training a small student model on the outputs of a larger teacher model. A soft target is the teacher's full probability row, and it carries more teaching than the single correct class.

Hard target

A hard target keeps only the winning class. The two losing classes receive the same 0.0.

logits = torch.tensor([[2.0, 1.0, -1.5]])
print("Hard target:", F.one_hot(logits.argmax(dim=1), num_classes=3).float().tolist())

Temperature softens the row

Dividing the scores by T before softmax flattens the row. The second-best class grows from 0.26 to 0.35.

print("Soft T=1:", [round(v, 2) for v in F.softmax(logits / 1.0, dim=1)[0].tolist()])
print("Soft T=4:", [round(v, 2) for v in F.softmax(logits / 4.0, dim=1)[0].tolist()])

Example

import torch
import torch.nn.functional as F
from torch import nn

torch.manual_seed(0)
teacher = nn.Linear(2, 3)
with torch.no_grad():
    teacher.weight.copy_(torch.tensor([[2.0, 0.0], [0.0, 2.0], [-1.0, -1.0]]))
    teacher.bias.copy_(torch.tensor([0.0, 0.0, 0.0]))
teacher.eval()

inputs = torch.tensor([[1.0, 0.5], [0.5, 1.0], [-1.0, 0.0]])
names = ["bar", "ring", "square"]
with torch.no_grad():
    logits = teacher(inputs)
    hard = F.one_hot(logits.argmax(dim=1), num_classes=3).float()
    soft1 = F.softmax(logits / 1.0, dim=1)
    soft4 = F.softmax(logits / 4.0, dim=1)

for i in range(3):
    print(f"Input {i}: Teacher scores {[round(v, 2) for v in logits[i].tolist()]}")
    print(f"  Hard target: {[round(v, 2) for v in hard[i].tolist()]}")
    print(f"  Soft target T=1: {[round(v, 2) for v in soft1[i].tolist()]}")
    print(f"  Soft target T=4: {[round(v, 2) for v in soft4[i].tolist()]}")
print("Teacher pick per input:", [names[v] for v in logits.argmax(dim=1).tolist()])

Key points

  • Teacher: the larger trained model that produces the targets
  • Student: the smaller model that learns from those targets
  • Hard target: a row with 1.0 on the correct class and 0.0 elsewhere
  • Soft target: a probability row that also grades the wrong classes
  • Temperature: a divisor applied to the scores before softmax, written T in this entry
  • A higher temperature makes the soft row flatter and exposes the smaller probabilities

In this example, the teacher is one Linear layer with hand-set weights and no training. Three fixed inputs run through it under no_grad. Each input prints its score row, its hard target, and two soft targets. At temperature 1, the first soft row reads 0.72, 0.26, and 0.02. At temperature 4, that row flattens to 0.46, 0.35, and 0.19. The hard target stays a single 1.0 with two zeros at every temperature. The soft rows grade the wrong classes, ranking ring above square for the first input. The teacher pick per input prints last, one class per input. The next lesson trains a student on rows like these.

Note

The teacher in the closing program has hand-set weights, so every printed number can be checked by hand

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

Exercise

What does raising the temperature do to a teacher's soft target row?