Nesterov Momentum makes one clever change to standard momentum: instead of computing the gradient at the current position and then applying momentum, it "looks ahead" to where momentum is about to carry it, and computes the gradient there instead.
Formula
Compare to standard momentum's \(\nabla L(\mathbf{w}_t)\): Nesterov instead evaluates the gradient at the "look-ahead" point \(\mathbf{w}_t - \beta\eta\mathbf{v}_{t-1}\) — an estimate of where the parameters will be after applying the existing momentum, before adding the new gradient's contribution.
The Intuition — A Smarter Ball
Standard momentum is like a ball that blindly keeps rolling based on its current velocity, then separately checks the slope where it currently is. Nesterov is like a more careful ball that first estimates where its current momentum will carry it, checks the slope there, and corrects its course accordingly — catching a change in slope slightly earlier than standard momentum would, since it's effectively looking one step ahead.
Why the Correction Matters
Because Nesterov evaluates the gradient at the look-ahead position, it can start correcting course before fully overshooting a minimum.
Code
import torch.optim as optim
# PyTorch's SGD supports Nesterov momentum directly via a flag
optimizer = optim.SGD([w], lr=0.1, momentum=0.9, nesterov=True)In Practice — A Small but Consistent Improvement
Nesterov momentum typically provides a modest improvement over standard momentum in convergence speed and stability, at essentially no additional computational cost — which is why it's commonly enabled by default (or nearly so) in many training recipes that use SGD with momentum.
Common Mistakes
- Assuming Nesterov momentum requires meaningfully different hyperparameter tuning from standard momentum — in practice, the same \(\beta\) and learning rate ranges that work for standard momentum are typically reasonable starting points for Nesterov as well.
Interview Relevance
Q: "What's the conceptual difference between standard momentum and Nesterov momentum?" Standard momentum computes the gradient at the current position, then combines it with the accumulated velocity. Nesterov momentum first estimates a "look-ahead" position based on the existing velocity, computes the gradient there instead, and uses that look-ahead gradient to update the velocity — effectively correcting course slightly earlier when momentum is about to overshoot.
Practice Question
Explain, in your own words, why evaluating the gradient at a "look-ahead" position can help an optimizer respond faster to an approaching minimum than evaluating it at the current position.
