PyTorch Deterministic Mode

Deterministic mode is a PyTorch setting that bans operations with no repeatable implementation. With it on, the same code on the same inputs must give the same result on every run.

Turn the mode on

One call turns the mode on for the whole program. A second call reports the current setting.

torch.use_deterministic_algorithms(True)
print("Enabled:", torch.are_deterministic_algorithms_enabled())

An operation it rejects

MaxUnpool2d rebuilds a full-size map from a pooled one and its stored indices. Deterministic mode refuses the operation outright.

torch.use_deterministic_algorithms(True)
pooled, indices = nn.MaxPool2d(2, return_indices=True)(torch.ones(1, 1, 4, 4))
try:
    nn.MaxUnpool2d(2)(pooled, indices, output_size=(1, 1, 4, 4))
    print("Rejected:", False)
except RuntimeError as error:
    print("Rejected:", "does not have a deterministic implementation" in str(error))

Warn instead of reject

warn_only=True lets the same operation run. PyTorch prints a warning to the error stream instead of raising.

torch.use_deterministic_algorithms(True, warn_only=True)
pooled, indices = nn.MaxPool2d(2, return_indices=True)(torch.ones(1, 1, 4, 4))
out = nn.MaxUnpool2d(2)(pooled, indices, output_size=(1, 1, 4, 4))
print("Ran with warning only:", tuple(out.shape))

Example

import torch
from torch import nn

torch.manual_seed(0)
torch.use_deterministic_algorithms(True)

x = torch.tensor([[1.0], [2.0], [3.0], [4.0]])
y = torch.tensor([[3.0], [5.0], [7.0], [9.0]])
model = nn.Linear(1, 1)
loss_fn = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.05)

losses = []
for step in range(3):
    loss = loss_fn(model(x), y)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    losses.append(round(loss.item(), 4))

print("Deterministic losses:", losses)
print("Mode enabled:", torch.are_deterministic_algorithms_enabled())

torch.manual_seed(0)
image = torch.randn(1, 1, 4, 4)
pooled, indices = nn.MaxPool2d(2, return_indices=True)(image)
try:
    nn.MaxUnpool2d(2)(pooled, indices, output_size=(1, 1, 4, 4))
    print("max_unpool2d: no error")
except RuntimeError as error:
    print("Rejected op: max_unpool2d")
    print("Error:", str(error).split(" You can turn off")[0])

Key points

Deterministic algorithm: an operation that always returns the same output for the same input
Use torch.use_deterministic_algorithms(True) to turn the mode on
An operation with no deterministic implementation raises a RuntimeError in this mode
warn_only=True downgrades that error to a warning, so the operation still runs
Deterministic implementations can be slower than the fastest ones
The mode trades operation choice for a guarantee that runs repeat

In this example, seed 0 is set and deterministic mode is turned on before anything runs. A one-layer model then trains for three steps on the line data. The three losses print as a trace, and the mode check prints True. Seed 0 is set again before a random image is drawn. MaxPool2d stores both the pooled map and the indices of the winners. MaxUnpool2d then tries to rebuild the full-size map, and deterministic mode rejects it. The program catches the RuntimeError and prints its first sentence. That sentence is the stable part of the error, and it names the missing deterministic implementation.

Note

In torch 2.14.1+cpu, max_unpool2d has no deterministic implementation, so the mode rejects it

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

import torch
from torch import nn

torch.manual_seed(0)
torch.use_deterministic_algorithms(True)

x = torch.tensor([[1.0], [2.0], [3.0], [4.0]])
y = torch.tensor([[3.0], [5.0], [7.0], [9.0]])
model = nn.Linear(1, 1)
loss_fn = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.05)

losses = []
for step in range(3):
    loss = loss_fn(model(x), y)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    losses.append(round(loss.item(), 4))

print("Deterministic losses:", losses)
print("Mode enabled:", torch.are_deterministic_algorithms_enabled())

torch.manual_seed(0)
image = torch.randn(1, 1, 4, 4)
pooled, indices = nn.MaxPool2d(2, return_indices=True)(image)
try:
    nn.MaxUnpool2d(2)(pooled, indices, output_size=(1, 1, 4, 4))
    print("max_unpool2d: no error")
except RuntimeError as error:
    print("Rejected op: max_unpool2d")
    print("Error:", str(error).split(" You can turn off")[0])

Exercise

What happens when deterministic mode meets an operation with no deterministic implementation?

It raises a RuntimeError It silently accepts a random result It moves the operation to a GPU It turns the mode off