PyTorch Second Derivatives
A second derivative is the derivative of a derivative. It measures curvature, which is how sharply the graph bends at a point.
The second call fails by default
The first derivative prints 32 at x equal to two. The last line raises a RuntimeError saying the tensor does not require grad and has no grad_fn.
import torch
x = torch.tensor([2.0], requires_grad=True)
y = x ** 4
first = torch.autograd.grad(y.sum(), x)[0]
print("First derivative:", first.tolist())
print("First needs grad:", first.requires_grad)
second = torch.autograd.grad(first.sum(), x)[0]create_graph keeps the gradient alive
create_graph=True records operations on the gradient itself. The second call now returns 48, the value of 12x squared at two.
x = torch.tensor([2.0], requires_grad=True)
y = x ** 4
first = torch.autograd.grad(y.sum(), x, create_graph=True)[0]
second = torch.autograd.grad(first.sum(), x)[0]
print("First derivative:", first.tolist())
print("Second derivative:", second.tolist())The gradient carries a graph
The first derivative still reports requires_grad as True here. Its grad_fn is present, which is the record the second call walks.
print("First derivative still tracked:", first.requires_grad)
print("First derivative has a graph:", first.grad_fn is not None)Example
import torch
x = torch.tensor([0.5, 1.0, 2.0], requires_grad=True)
y = x ** 4
first = torch.autograd.grad(y.sum(), x, create_graph=True)[0]
second = torch.autograd.grad(first.sum(), x)[0]
print("Points:", x.detach().tolist())
print("Values x**4:", y.detach().tolist())
print("First derivative 4x**3:", first.detach().tolist())
print("Second derivative 12x**2:", second.detach().tolist())Key points
- Curvature: how sharply a graph bends at a point
- Use create_graph=True on autograd.grad to keep the first gradient tracked
- A tracked first gradient can go through autograd.grad a second time
- Without create_graph, the first gradient carries no graph and the second call fails
- For x to the fourth, the first derivative is 4x cubed and the second is 12x squared
- In short, create_graph makes the gradient itself a tracked result
In this example, x holds three points and y raises each one to the fourth power. The first autograd.grad call passes create_graph=True, so the gradient stays tracked. Summing first gives one gradient row for the three points. The first derivative prints 0.5, 4.0, and 32.0, matching 4x cubed. The second call differentiates that row and prints 3.0, 12.0, and 48.0. Those values match 12x squared at the same three points. The middle point shows the bend plainly, with slope 4 and curvature 12.
Note
Second derivatives cost a second graph, so plain training leaves create_graph off
Try it Yourself
Edit the code, then press Run.
Running is simulated · Python Playground
Exercise
What does create_graph=True change about the first gradient?