🎓 A1Learner
🔍

PyTorch Reading the Results

A per-class report counts the right and wrong answers inside each class. One accuracy number for the whole dataset can hide a class the model keeps missing.

Predictions next to labels

Comparing the two tensors gives one True or False per row.

labels = torch.tensor([0, 1, 2, 1, 0])
predictions = torch.tensor([0, 2, 2, 1, 1])
print(predictions == labels)

Count one class's results

The mask picks the rows of class 2. Only those rows are counted.

labels = torch.tensor([0, 0, 0, 1, 1, 2, 2, 2])
predictions = torch.tensor([0, 1, 0, 1, 2, 2, 0, 2])
mask = labels == 2
print(int(mask.sum()))
print(int((predictions[mask] == 2).sum()))

Look at the mistakes

The wrong rows print with their true labels and the model's answers.

labels = torch.tensor([0, 1, 2, 1, 0])
predictions = torch.tensor([0, 2, 2, 1, 1])
wrong = predictions != labels
print(int(wrong.sum()))
print(labels[wrong])
print(predictions[wrong])

Example

import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset

torch.manual_seed(0)

per_class = 50
centres = [torch.tensor([-1.5, 0.0]), torch.tensor([1.5, 0.0]),
           torch.tensor([0.0, 1.5])]
blobs = [torch.randn(per_class, 2) * 1.0 + centre for centre in centres]
points = torch.cat(blobs)
labels = torch.cat([torch.full((per_class,), i, dtype=torch.long)
                     for i in range(3)])

dataset = TensorDataset(points, labels)
loader = DataLoader(dataset, batch_size=32, shuffle=True)

torch.manual_seed(1)
model = nn.Sequential(
    nn.Linear(2, 16),
    nn.ReLU(),
    nn.Linear(16, 3),
)
loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.05)

for epoch in range(30):
    for batch_points, batch_labels in loader:
        scores = model(batch_points)
        loss = loss_fn(scores, batch_labels)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

model.eval()
with torch.no_grad():
    predictions = model(points).argmax(dim=1)

correct = int((predictions == labels).sum())
accuracy = correct / labels.shape[0] * 100
print(f"Overall accuracy: {accuracy:.0f}%")
print("Per-class report:")
for class_id in range(3):
    mask = labels == class_id
    total = int(mask.sum())
    right = int((predictions[mask] == class_id).sum())
    print(f"Class {class_id}: {right} of {total} right, "
          f"{total - right} mistakes")

Key points

  • Accuracy: the fraction of all rows answered right
  • Per-class report: the right and wrong counts for one class at a time
  • Use a mask like labels == 2 to pick out one class's rows
  • A class can fail often while the overall accuracy still looks fine

In this example, three blobs overlap on purpose, so the model cannot get every row right. The model trains the same way as the three-blob classifier, with batches from a DataLoader. The overall accuracy prints first, and it reads 84%. Then the report counts each class on its own. Class 1 loses only 3 rows. Classes 0 and 2 lose 10 and 11 rows. The single accuracy number hid that gap.

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

Exercise

A model scores 90% overall, but class 2 is right only half the time. What shows this?