🎓 A1Learner
🔍

PyTorch Text as Indices

Text as indices is a way of storing words as numbers from a fixed word list. A vocabulary assigns each word one index, and a sentence becomes a list of those numbers.

The vocabulary

The list starts with <pad> and then the 17 review words. Its length is the vocabulary size.

print(vocab[:5])
print(len(vocab))

One word, one number

The dictionary answers with the word's position in the list. warm sits at 5 and dull at 7.

print(word_to_index["warm"], word_to_index["dull"])

Encode, then decode

Encoding swaps each word for its index. Decoding swaps them back, so the round trip returns the same sentence.

indices = encode("the warm bright scene")
print(indices)
print(decode(indices))

Example

positive = ["great", "lovely", "bright", "fun", "warm"]
negative = ["awful", "dull", "cold", "boring", "grim"]
filler = ["the", "a", "movie", "story", "scene", "really", "very"]

vocab = ["<pad>"] + positive + negative + filler
word_to_index = {word: index for index, word in enumerate(vocab)}
index_to_word = {index: word for word, index in word_to_index.items()}


def encode(text):
    return [word_to_index[word] for word in text.split()]


def decode(indices):
    return " ".join(index_to_word[index] for index in indices)


reviews = ["a lovely warm story", "the dull cold scene", "really great fun movie"]
print("Vocabulary size:", len(vocab))
for text in reviews:
    indices = encode(text)
    print(text, "->", indices, "->", decode(indices))

Key points

  • Vocabulary: the fixed list of words a model can use, each with its own index
  • Index 0 is kept for <pad>, a placeholder entry that means no word
  • Use a dictionary to map each word to its index
  • Decoding walks the indices back through the reversed map
  • A word outside the vocabulary has no index

In this example, the vocabulary joins a pad entry, five positive words, five negative words, and seven filler words. The size prints as 18. Each review encodes to a list of indices and decodes straight back to the same words. The printed lines show both directions for three tiny reviews. A model never sees the letters of a word. It sees only the number, and the number keeps the word's identity.

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

Exercise

Why does index 0 hold <pad> instead of a real word?