🎓 A1Learner
🔍

PyTorch RNN Basics

A recurrent neural network reads a sequence one step at a time and carries a hidden state along. The hidden state is the network's memory of the steps so far.

A layer with no memory

A plain layer call takes one input and returns one output. No state passes between two calls, so step two cannot see step one.

layer = nn.Linear(1, 3)
out = layer(torch.tensor([[1.0]]))

The recurrent module

The module takes an input size and a hidden size. Hidden size 3 keeps the printed states short.

rnn = nn.RNN(input_size=1, hidden_size=3, batch_first=True)
print(rnn)

The state starts at zero

The starting state is all zeros. Its shape holds layers, batch, and hidden size.

h0 = torch.zeros(1, 1, 3)
print(tuple(h0.shape))

One step in, one new state

One call with one step returns a changed state. The state now holds a trace of that step.

seq = torch.tensor([[[1.0]]])
out, h1 = rnn(seq, h0)
print([round(v, 3) for v in h1[0, 0].tolist()])

Example

import torch
from torch import nn

torch.manual_seed(0)
rnn = nn.RNN(input_size=1, hidden_size=3, batch_first=True)

seq = torch.tensor([[[0.0], [1.0], [0.5], [-1.0]]])
print("Input shape:", tuple(seq.shape))

h = torch.zeros(1, 1, 3)
with torch.no_grad():
    for step in range(seq.shape[1]):
        out, h = rnn(seq[:, step:step + 1], h)
        state = [round(v, 3) for v in h[0, 0].tolist()]
        print("Step", step + 1, "input", seq[0, step, 0].item(), "hidden:", state)

    full_out, full_h = rnn(seq)
    print("Whole-sequence final hidden:", [round(v, 3) for v in full_h[0, 0].tolist()])
    print("Same final state:", torch.allclose(h, full_h))

Key points

  • Hidden state: a small vector the network updates at every step
  • Use nn.RNN to build a plain recurrent layer
  • The state starts as zeros before the first step
  • Each step mixes the current input with the state from the last step
  • The output at a step is the new hidden state itself
  • Feeding steps one by one and feeding the whole sequence give the same final state
  • Hidden state is the memory, restated plainly: later steps can use what earlier steps left in it

In this example, seed 0 builds an RNN with input size 1 and hidden size 3. The sequence holds four steps with values 0.0, 1.0, 0.5, and -1.0. The loop feeds one step at a time and keeps the returned state. The state starts at zeros and changes at every step. Step 2 reads input 1.0 plus the state that step 1 left behind. The whole sequence then goes through in one call. Its final hidden state matches the step-by-step final state exactly. That match is the point of the lesson. The carried state is the only path by which early steps shape later answers.

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

Exercise

What does the hidden state hold after step 3 of a sequence?