🎓 A1Learner
🔍

PyTorch Multi-Label Losses

Multi-label classification gives each row several yes-or-no answers at once. One score per label comes out of the model, and BCEWithLogitsLoss scores all of them together.

Labels are a table

Each row is one document. Each column is one label, and 1 means yes.

import torch

labels = torch.tensor([[1.0, 0.0, 1.0],
                       [0.0, 1.0, 0.0]])
print(labels.shape)
print(labels[0])

One probability per label

Sigmoid scores each logit on its own. The three probabilities do not need to sum to 1.

import torch

logits = torch.tensor([[2.0, -1.0, 0.5]])
print(torch.sigmoid(logits).round(decimals=4))

One loss over all labels

BCEWithLogitsLoss takes the full logit table and the full label table in one call.

import torch

logits = torch.tensor([[2.0, -1.0, 0.5],
                       [0.0, 1.0, -2.0]])
labels = torch.tensor([[1.0, 0.0, 1.0],
                       [0.0, 1.0, 0.0]])
loss_fn = torch.nn.BCEWithLogitsLoss()
print(f"{loss_fn(logits, labels).item():.4f}")

Example

import torch

torch.manual_seed(0)

words = ["cat", "dog", "fish", "bird", "swim",
         "fly", "run", "sleep", "bark", "sing"]
# Each document is a row of word counts. Each label reads fixed words.
# pet: cat or dog. water: fish or swim. air: bird or fly.
docs = (torch.rand(120, 10) < 0.3).float()
labels = torch.stack([
    ((docs[:, 0] + docs[:, 1]) > 0).float(),
    ((docs[:, 2] + docs[:, 4]) > 0).float(),
    ((docs[:, 3] + docs[:, 5]) > 0).float(),
], dim=1)

model = torch.nn.Linear(10, 3)
optimizer = torch.optim.Adam(model.parameters(), lr=0.1)
loss_fn = torch.nn.BCEWithLogitsLoss()

for step in range(300):
    loss = loss_fn(model(docs), labels)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

with torch.no_grad():
    pred = (torch.sigmoid(model(docs)) >= 0.5).float()
    names = ["pet", "water", "air"]
    for i, name in enumerate(names):
        acc = (pred[:, i] == labels[:, i]).float().mean().item()
        print(f"Label {name}: accuracy {acc:.3f}")
print(f"Final loss: {loss.item():.4f}")

Key points

  • Multi-label: each row carries several labels, and each label is its own yes-or-no question
  • Multi-class: each row carries exactly one label out of several
  • Use nn.BCEWithLogitsLoss when the labels are separate yes-or-no questions
  • The model outputs one logit per label, so an output row has one score per label
  • Sigmoid turns each logit into its own probability, separate from the other labels
  • A threshold of 0.5 on each probability turns the scores into yes-or-no answers
  • Separate questions get separate scores, and one loss scores them all at once

In this example, each document is a row of word counts over ten words. Three labels read fixed words from each row. The pet label is yes when cat or dog appears. The water label is yes when fish or swim appears. The air label is yes when bird or fly appears. A linear model outputs three logits per document. BCEWithLogitsLoss scores all three labels in one call. After 300 steps the final loss is 0.0107. The per-label accuracy prints 1.000 for pet, 1.000 for water, and 1.000 for air. Each label learned its own pair of words.

Note

Softmax is wrong here, because softmax forces the labels to compete for one shared total

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

Exercise

A dataset gives each photo three separate yes-or-no tags. Which loss fits the task?