🎓 A1Learner
🔍

PyTorch Gradients Accumulate

Gradients accumulate, which means each backward() call adds its result into .grad. The stored gradient grows until it is cleared on purpose.

One backward, one gradient

The first call stores the gradient of x ** 2 at x = 3. The stored value is 6.

import torch

x = torch.tensor(3.0, requires_grad=True)
y = x ** 2
y.backward()
print(x.grad.item())

A second backward adds more

The calculation is built fresh and backward() runs again. The stored gradient doubles to 12.

y = x ** 2
y.backward()
print(x.grad.item())

zero_() clears the total

zero_() writes zeros into the stored gradient in place. The next backward() starts from zero again.

x.grad.zero_()
print(x.grad.item())

Example

import torch

x = torch.tensor(3.0, requires_grad=True)
y = x ** 2
y.backward()
print("After one backward:", x.grad.item())
y = x ** 2
y.backward()
print("After two backwards:", x.grad.item())
x.grad.zero_()
y = x ** 2
y.backward()
print("After zero and one backward:", x.grad.item())

Key points

  • backward() adds into .grad instead of replacing it
  • Two backward() calls on the same maths double the stored gradient
  • A fresh calculation is built before each backward() call
  • Use .grad.zero_() to clear a stored gradient back to zero
  • In short, .grad is a running total, and zero_() resets the total

In this example, x is tracked at 3.0 and y squares it. The first backward() stores the gradient 6.0 in x.grad. y is then built again from x, because the first record was used up. The second backward() adds another 6.0, so x.grad prints 12.0. zero_() clears the stored gradient. One more fresh squaring and backward() brings the print back to 6.0.

Note

Training loops clear gradients every round for exactly this reason

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

Exercise

What does x.grad.zero_() do?