🎓 A1Learner
🔍

PyTorch L-BFGS

L-BFGS is an optimizer that estimates curvature from its recent steps. Curvature is how sharply the loss bends around the current point.

The closure recomputes the loss

The closure clears old gradients and runs backward each time it is called. One call here reports the starting loss of a small bowl.

import torch

params = torch.tensor([3.0, -2.0], requires_grad=True)
optimizer = torch.optim.LBFGS([params], lr=1.0, max_iter=20)

def closure():
    optimizer.zero_grad()
    loss = (params[0] - 1.0) ** 2 + 2.0 * (params[1] + 1.0) ** 2
    loss.backward()
    return loss

print(f"{closure().item():.4f}")

One parameter, one step call

A single step call moves one parameter straight to the bottom of its bowl.

import torch

p = torch.tensor([4.0], requires_grad=True)
optimizer = torch.optim.LBFGS([p], lr=1.0, max_iter=20)

def closure():
    optimizer.zero_grad()
    loss = (p - 1.0) ** 2
    loss.backward()
    return loss

optimizer.step(closure)
print(f"{p.item():.4f}")

Counting the hidden evaluations

A counter inside the closure shows how much work one step call hides. The count here covers the same one-parameter bowl.

import torch

p = torch.tensor([4.0], requires_grad=True)
optimizer = torch.optim.LBFGS([p], lr=1.0, max_iter=20)
calls = 0

def closure():
    global calls
    calls += 1
    optimizer.zero_grad()
    loss = (p - 1.0) ** 2
    loss.backward()
    return loss

optimizer.step(closure)
print(calls)

Example

import torch

torch.manual_seed(0)
params = torch.tensor([3.0, -2.0], requires_grad=True)
optimizer = torch.optim.LBFGS([params], lr=1.0, max_iter=20)
evals = 0

def closure():
    global evals
    evals += 1
    optimizer.zero_grad()
    loss = (params[0] - 1.0) ** 2 + 2.0 * (params[1] + 1.0) ** 2
    loss.backward()
    return loss

print(f"Start loss: {closure().item():.4f}")
evals = 0
optimizer.step(closure)
final = (params[0] - 1.0) ** 2 + 2.0 * (params[1] + 1.0) ** 2
print(f"Closure evaluations during step: {evals}")
print(f"Final x: {params[0].item():.4f}")
print(f"Final y: {params[1].item():.4f}")
print(f"Final loss: {final.item():.6f}")

Key points

  • Second-order method: a method that uses curvature, not only the slope
  • Smooth: the loss bends gently, with no sharp corners or jumps
  • Closure: a function that recomputes the loss and its gradients when called
  • Use optimizer.step(closure) to run L-BFGS, passing the closure itself
  • One step call may run the closure many times while it searches
  • L-BFGS fits small and smooth problems, where each loss call stays cheap

In this example, the loss is a quadratic bowl. Its bottom sits at x of 1 and y of minus 1. The parameters start at 3 and minus 2, where the loss is 6. The closure counts its own calls, then the counter resets before the real step. One step call runs the closure five times while it searches. The parameters land on the bottom of the bowl. The final loss prints as 0.000000 at six decimals.

Note

L-BFGS keeps a short history of past steps in memory to estimate curvature

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

Exercise

Why does L-BFGS take a closure instead of one loss value?