🎓 A1Learner
🔍

PyTorch Padding Sequences

Padding sequences is the step that makes sentences of different lengths fit one tensor. Shorter sentences are filled with a padding value until every row has the same length.

The pain: stacking ragged rows fails

torch.stack refuses with a RuntimeError. The message says the tensors must be equal size, which ragged sentences are not.

first = torch.tensor([12, 2, 5, 14])
second = torch.tensor([16, 1, 4])
torch.stack([first, second])

pad_sequence fills the gaps

The shorter row gains a zero at its end. Both rows now share one length, so they form a batch.

padded = pad_sequence([torch.tensor([1, 2]), torch.tensor([3, 4, 5])],
                       batch_first=True, padding_value=0)
print(tuple(padded.shape))
print(padded)

batch_first flips the axes

Without batch_first, the length comes first in the shape. The same two sentences give (4, 2) instead of (2, 4).

sentences = [first, second]
print(tuple(pad_sequence(sentences, padding_value=0).shape))
print(tuple(pad_sequence(sentences, batch_first=True, padding_value=0).shape))

The padding index in an embedding

An embedding with padding_idx=0 holds the pad row at zero during training. A padded batch feeds the layer like any other batch.

embedding = nn.Embedding(18, 4, padding_idx=0)
batch = pad_sequence(sentences, batch_first=True, padding_value=0)
print(embedding(batch).shape)
print(embedding.padding_idx)

Example

import torch
from torch.nn.utils.rnn import pad_sequence

sentences = [
    torch.tensor([12, 2, 5, 14]),
    torch.tensor([11, 7, 8, 15]),
    torch.tensor([16, 1, 4]),
]

for sentence in sentences:
    print("Length", len(sentence), ":", sentence.tolist())

padded = pad_sequence(sentences, batch_first=True, padding_value=0)
print("Padded batch shape:", tuple(padded.shape))
print(padded)

Key points

  • Padding index: the index that means no word, here 0 for <pad>
  • Use pad_sequence to fill shorter rows up to the longest row
  • batch_first=True gives a shape of (batch, length) instead of the default (length, batch)
  • The padded cells hold the padding value, and later steps must ignore them
  • pad_sequence takes a list of tensors whose lengths may differ

In this example, three sentences from the tiny reviews appear as index lists of lengths 4, 4, and 3. The lengths print first, one line per sentence. pad_sequence then fills the short row with one zero, the padding index. The batch shape prints as (3, 4). The padded tensor shows the zero at the end of the last row. Every row now has one length, so the batch can feed a model in a single call.

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

Exercise

What does pad_sequence put into the empty cells of a short row?