🎓 A1Learner
🔍

PyTorch Bidirectional RNNs

A bidirectional RNN reads a sequence forward and backward with two separate passes. Each step's output joins what came before it and what comes after it.

One way sees half the story

A forward-only pass at step 2 knows steps 1 and 2. It knows nothing about steps 3 to 6.

seq = torch.zeros(1, 6, 1)
one_way = nn.RNN(input_size=1, hidden_size=4, batch_first=True)
out1, h1 = one_way(seq)
print(tuple(out1.shape), tuple(h1.shape))

Both ways at once

The second pass reads the sequence in reverse. Each output row now holds four forward numbers and four backward numbers.

two_way = nn.RNN(input_size=1, hidden_size=4, batch_first=True, bidirectional=True)
out2, h2 = two_way(seq)
print(tuple(out2.shape), tuple(h2.shape))

Example

import torch
from torch import nn

torch.manual_seed(0)
values = [0.0, 0.5, 1.0, 0.5, 0.0, -0.5]
labels = ["flat", "rising", "peak", "falling", "flat", "falling"]
seq = torch.tensor(values).reshape(1, 6, 1)

rnn = nn.RNN(input_size=1, hidden_size=4, batch_first=True, bidirectional=True)
with torch.no_grad():
    output, hidden = rnn(seq)

print("Step labels:", labels)
print("Input shape:", tuple(seq.shape))
print("Output shape:", tuple(output.shape))
print("Hidden shape:", tuple(hidden.shape))
print("Hidden layers x directions:", hidden.shape[0])
print("Output features per step:", output.shape[2])
print("First step output:", [round(v, 3) for v in output[0, 0].tolist()])

Key points

  • Forward pass: reads the steps in time order and builds one hidden state per step
  • Backward pass: reads the steps in reverse order and builds a second hidden state per step
  • Output features double, since each step joins both directions
  • Hidden shape gains a directions axis, so one layer gives shape (2, batch, hidden)
  • Use bidirectional=True on nn.RNN, nn.LSTM, or nn.GRU to turn on the second pass
  • A bidirectional model fits labelling tasks where the whole sequence is known

In this example, six values rise to a peak and fall again. The labels name each step, from flat and rising through peak and back to falling. A bidirectional RNN with hidden size 4 reads the sequence. The input shape is (1, 6, 1). The output shape is (1, 6, 8), since each step joins four forward numbers with four backward numbers. The hidden shape is (2, 1, 4), one final state per direction. The first step's output prints all eight numbers. Its backward half already carries the peak and the fall, which a forward pass cannot know at step one. That is the strength and the limit of the design. It suits labelling a recorded sequence, never forecasting one.

Note

A bidirectional pass cannot predict the future, since the backward pass reads values that have not happened yet

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

Exercise

Why can a bidirectional RNN not forecast the next value of a live stream?