🎓 A1Learner
🔍

PyTorch Stopping Gradients

torch.no_grad() is a block where autograd records nothing. Work inside the block runs as plain tensor maths, with no record kept.

Tracked work keeps a record

Outside any block, tracking is on. The result reports that it needs gradients.

import torch

x = torch.tensor(3.0, requires_grad=True)
y = x ** 2
print(y.requires_grad)

The same work under no_grad()

Inside the block, nothing is recorded. The result needs no gradients and names no grad_fn.

with torch.no_grad():
    z = x ** 2
print(z.requires_grad)
print(z.grad_fn)

Detach one tensor

detach() returns a tensor outside the graph. It keeps the same values, with tracking off.

base = torch.tensor(4.0, requires_grad=True)
squared = base ** 2
plain = squared.detach()
print(plain.requires_grad)
print(plain.item())

Example

import torch

x = torch.tensor(3.0, requires_grad=True)
y = x ** 2
y.backward()
print("Tracked gradient:", x.grad.item())

fresh = torch.tensor(3.0, requires_grad=True)
with torch.no_grad():
    z = fresh ** 2
print("Untracked result:", z.item())
print("Untracked needs grad:", z.requires_grad)
print("Gradient after no_grad:", fresh.grad)

Key points

  • Use torch.no_grad() around work that needs no gradients
  • A result made under no_grad() has requires_grad set to False
  • No backward() can run on an untracked result
  • Graph: the record of operations autograd keeps for a result
  • Use detach() to take one tensor out of the graph
  • In short, no_grad() stops recording for a block and detach() stops it for one tensor

In this example, x is tracked and y squares it. backward() fills x.grad, and the tracked gradient prints as 6.0. fresh then repeats the same squaring under no_grad(). The untracked result holds 9.0, but its requires_grad prints False. Nothing was recorded, so fresh.grad stays None.

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

Exercise

What does a calculation inside torch.no_grad() produce?