🎓 A1Learner
🔍

PyTorch requires_grad

requires_grad is a flag on a tensor that switches tracking on. A tensor with the flag set hands autograd a record of every operation on it.

Switch tracking on

The flag prints True, so autograd tracks this tensor. A directly created tensor is a leaf.

import torch

x = torch.tensor(2.0, requires_grad=True)
print(x.requires_grad)
print(x.is_leaf)

One operation, one name

The result is not a leaf, because an operation made it. Its print names MulBackward0 as the last operation.

y = x * 3
print(y)
print(y.is_leaf)

A second step adds a second name

The new result names AddBackward0. Each result remembers only its own last operation.

z = y + 1
print(z)

Example

import torch

x = torch.tensor(2.0, requires_grad=True)
step1 = x * 3
step2 = step1 + 1
print("x is leaf:", x.is_leaf)
print("step1:", step1)
print("step2:", step2)
print("step2 is leaf:", step2.is_leaf)

Key points

  • Use requires_grad=True when a tensor is created to start tracking
  • Leaf tensor: a tensor created directly, not made by an operation
  • A result tensor gets its tracking from its inputs
  • grad_fn: the record on a result tensor that names its last operation
  • .is_leaf is True for a leaf tensor and False for a result
  • Tracking starts at the leaf, and each operation adds one named step

In this example, x is created with requires_grad=True, so x is a tracked leaf. step1 multiplies x by 3 and holds 6. Its print names MulBackward0 as its grad_fn. step2 adds 1 to step1 and holds 7. Its grad_fn names AddBackward0, the last operation. The final print shows step2 is not a leaf, because an operation made it.

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

Exercise

Which tensor is a leaf?