🎓 A1Learner
🔍

PyTorch Reading Text Classifier Errors

An error reading is a close look at the exact inputs a model gets wrong. The word lists behind each mistake show why the model failed.

A word training never used

The word not joins the vocabulary for the tricky set. No training review contains it, so its embedding never trains.

print(WORD_ID["not"], len(WORDS))

Order cannot change the answer

A bag model averages word vectors. The average is the same in any word order.

pair = [(["great", "awful"], 0), (["awful", "great"], 0)]
xp, _ = encode(pair, 6)
probs = torch.softmax(model(xp), dim=1)
print([round(v, 3) for v in probs[0].tolist()])
print([round(v, 3) for v in probs[1].tolist()])

Confidence behind one mistake

Softmax gives the negative class probability 1.0 here. The model is fully sure and fully wrong.

one = [(["not", "awful", "dull"], 1)]
xo, _ = encode(one, 6)
p = torch.softmax(model(xo), dim=1)
print([round(v, 3) for v in p[0].tolist()])

Example

import torch
from torch import nn

POSITIVE = ["great", "lovely", "bright", "fun", "warm"]
NEGATIVE = ["awful", "dull", "cold", "boring", "grim"]
FILLER = ["the", "a", "movie", "story", "scene", "really", "very"]
WORDS = ["<pad>"] + POSITIVE + NEGATIVE + FILLER + ["not"]
WORD_ID = {word: i for i, word in enumerate(WORDS)}


def make_review(gen):
    label = int(torch.randint(0, 2, (1,), generator=gen))
    side = POSITIVE if label == 1 else NEGATIVE
    n_sent = int(torch.randint(2, 4, (1,), generator=gen))
    n_fill = int(torch.randint(1, 4, (1,), generator=gen))
    picks = torch.randperm(len(side), generator=gen)[:n_sent]
    words = [side[i] for i in picks.tolist()]
    picks = torch.randint(0, len(FILLER), (n_fill,), generator=gen)
    words += [FILLER[i] for i in picks.tolist()]
    order = torch.randperm(len(words), generator=gen)
    return [words[i] for i in order.tolist()], label


def encode(reviews, width):
    rows = []
    for words, label in reviews:
        ids = [WORD_ID[w] for w in words]
        ids += [0] * (width - len(ids))
        rows.append((ids, label))
    x = torch.tensor([r[0] for r in rows])
    y = torch.tensor([r[1] for r in rows])
    return x, y


class BagClassifier(nn.Module):
    def __init__(self, vocab_size, dim):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, dim, padding_idx=0)
        self.linear = nn.Linear(dim, 2)

    def forward(self, ids):
        mask = (ids != 0).float().unsqueeze(-1)
        summed = (self.embedding(ids) * mask).sum(dim=1)
        counts = mask.sum(dim=1).clamp(min=1)
        return self.linear(summed / counts)


gen = torch.Generator().manual_seed(7)
reviews = [make_review(gen) for _ in range(200)]
width = 6
x_train, y_train = encode(reviews[:160], width)

torch.manual_seed(0)
model = BagClassifier(len(WORDS), 8)
loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.05)
for epoch in range(60):
    loss = loss_fn(model(x_train), y_train)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

# Tricky held-out reviews. Rule one: "not" flips the whole review.
# Rule two: in a mixed review, the last sentiment word wins.
tricky = [
    (["not", "great", "the", "movie"], 0),
    (["not", "lovely", "warm", "story"], 0),
    (["not", "fun", "bright", "really"], 0),
    (["not", "great", "lovely", "warm"], 0),
    (["not", "awful", "dull"], 1),
    (["not", "cold", "grim", "movie"], 1),
    (["not", "boring", "the", "story"], 1),
    (["not", "awful", "cold", "grim"], 1),
    (["great", "lovely", "awful"], 0),
    (["warm", "bright", "grim"], 0),
    (["fun", "lovely", "cold"], 0),
    (["awful", "dull", "great"], 1),
    (["grim", "cold", "lovely"], 1),
    (["boring", "dull", "warm"], 1),
]
x_test, y_test = encode(tricky, width)

names = {1: "positive", 0: "negative"}
with torch.no_grad():
    probs = torch.softmax(model(x_test), dim=1)
    pred = probs.argmax(dim=1)

mistakes = []
for i, (words, label) in enumerate(tricky):
    if pred[i].item() != label:
        mistakes.append((probs[i, pred[i]].item(), words, label, pred[i].item()))
mistakes.sort(reverse=True)

print("Tricky reviews:", len(tricky))
print("Mistakes:", len(mistakes))
print("Five most confident mistakes:")
for conf, words, label, guess in mistakes[:5]:
    print(" ".join(words), "| true:", names[label], "| predicted:", names[guess],
          "| confidence:", round(conf, 3))

Key points

  • A confident mistake is a wrong answer with a high score on the wrong side
  • Softmax turns two scores into two probabilities that add to one
  • The word not flips a review, and the bag model never saw it in training
  • A mixed review holds sentiment words from both sides
  • The bag average cannot see which sentiment word came last
  • Reading errors by their word lists shows the model's blind spots directly

In this example, the same generator and model from the last lesson train in the same way. The tricky set then breaks the dataset rules on purpose. Eight reviews start with not, which flips the whole review. Six mixed reviews hold both sides, and the last sentiment word sets the label. The model misses 13 of the 14 tricky reviews. The five most confident mistakes all print with confidence 1.0 after rounding. The top line reads not awful dull, where the bag sees two strong negative words and misses the flip. The fourth line, grim cold lovely, shows the mixed case, where the bag counts two negatives against one positive. The final line, awful dull great, fails the same way. A bag of words has no slot for order or for one small word that flips the rest. Sequence models, the next entry's topic, read words in order.

Note

Every mistake here is a data pattern the model cannot represent, not a training bug

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

Exercise

Why does the bag model fail on the review not awful dull?