Exponential decay shrinks the learning rate smoothly and continuously every single step, rather than in the sudden jumps of step decay — trading step decay's simplicity for a gentler, gradually-shrinking curve.
Formula
\(\gamma\) (e.g. 0.95, 0.99) is a decay rate applied every single epoch (or step) — unlike step decay's \(\gamma\), which only applies at discrete intervals.
Numerical Example
With \(\eta_0=0.1\), \(\gamma=0.95\): epoch 0: \(0.1\); epoch 10: \(0.1\times0.95^{10}\approx0.0599\); epoch 30: \(0.1\times0.95^{30}\approx0.0215\) — a smooth, continuous shrinkage rather than step decay's flat-then-drop pattern.
Graph
A smooth, continuously decreasing curve — steep early, flattening as it approaches zero.
Code
import torch.optim as optim
from torch.optim.lr_scheduler import ExponentialLR
optimizer = optim.SGD(model.parameters(), lr=0.1)
scheduler = ExponentialLR(optimizer, gamma=0.95)
for epoch in range(30):
# ... training loop for this epoch ...
scheduler.step()
print(scheduler.get_last_lr())Step Decay vs Exponential Decay
| Step Decay | Exponential Decay | |
|---|---|---|
| Shape | Flat segments, sudden drops | Smooth, continuous shrinkage |
| Interpretability | Very easy to reason about ("halves every 10 epochs") | Slightly less intuitive, but avoids abrupt loss-curve jumps |
| Hyperparameters | Step size \(s\) and decay factor \(\gamma\) | Just decay rate \(\gamma\) |
Common Mistakes
- Choosing a \(\gamma\) too close to 1 (e.g. 0.999) expecting meaningful decay — over a typical training run, this decays extremely slowly and may barely differ from a constant learning rate in practice.
- Choosing a \(\gamma\) too far from 1 (e.g. 0.7) — the learning rate can shrink to near-zero within just a handful of epochs, effectively freezing training prematurely.
Interview Relevance
Q: "When might you prefer exponential decay over step decay?" When you want a smoother loss curve without the small, temporary jumps that can appear right at step decay's sudden drop points — exponential decay changes the learning rate a tiny amount every single step instead of in occasional large jumps, at the cost of being slightly less intuitive to reason about directly.
Practice Question
With \(\eta_0=0.05\) and \(\gamma=0.9\), what is the learning rate at epoch 5? (You can approximate \(0.9^5 \approx 0.59\).)
