One Cycle Learning Rate

The One Cycle policy (Leslie Smith) combines warmup and decay into a single deliberate cycle spanning the entire training run — rising to a peak, then falling below even the original starting rate — and was shown to enable dramatically faster training ("super-convergence") in some settings.

The Shape of One Cycle

Three Phases

Momentum Often Cycles Inversely

The One Cycle policy commonly cycles momentum in the opposite direction — decreasing momentum while the learning rate rises (allowing the larger steps to be more directly gradient-driven), and increasing momentum as the learning rate falls (letting the optimizer's accumulated direction stabilize convergence during the careful final phase).

Code

import torch.optim as optim
from torch.optim.lr_scheduler import OneCycleLR

optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
total_steps = 1000
scheduler = OneCycleLR(
    optimizer,
    max_lr=0.1,
    total_steps=total_steps,
    pct_start=0.3   # 30% of training spent in the "rise" phase
)

for step in range(total_steps):
    # ... forward pass, loss, backward pass, optimizer.step() ...
    scheduler.step()   # called every STEP, not every epoch, for OneCycleLR specifically

Unlike the other schedules in this category, OneCycleLR is designed to be stepped once per training batch, not once per epoch — because its phases are defined relative to the total number of training steps across the entire run.

Why "Super-Convergence"

Leslie Smith's original research found that One Cycle, using a peak learning rate much higher than what would typically be considered "safe," could train some models to strong accuracy in dramatically fewer total epochs than a conventional fixed or gradually-decaying schedule — essentially trading a brief, controlled period of aggressive, higher-risk learning for a large net time savings, bookended by careful warmup and fine-tuning phases that keep the aggressive middle phase from destabilizing training.

Common Mistakes

• Calling scheduler.step() once per epoch for OneCycleLR (as is correct for most other schedulers) instead of once per batch — this desynchronizes the schedule from its intended per-step design and produces incorrect learning rate values throughout training.
• Choosing max_lr without first estimating a reasonable range (e.g. via a learning-rate range test) — One Cycle's benefits depend on the peak being aggressive but not so large that training destabilizes during the rise phase.

Interview Relevance

Q: "What makes the One Cycle learning rate policy different from a standard warmup-then-decay schedule?" One Cycle deliberately pushes the peak learning rate higher than typical warmup-then-decay schedules would use, and often cycles momentum inversely alongside it — the combination of an aggressive peak phase (for fast progress) with careful bookending rise and final-annealing phases (for stability) is what enables the "super-convergence" effect: strong final accuracy in fewer total epochs than more conservative schedules.

Key Takeaways — Optimization & LR Scheduling

• Batch, stochastic and mini-batch gradient descent differ only in how much data informs each update — mini-batch is the practical default everywhere.
• Momentum smooths the update direction; Nesterov improves it further by looking ahead before computing the gradient.
• AdaGrad introduced per-parameter adaptive rates but shrinks them irreversibly; RMSProp fixes this with a decaying average; Adam adds momentum plus bias correction on top of RMSProp's idea; AdamW fixes Adam's broken interaction with weight decay.
• A fixed learning rate is a compromise between early speed and late precision — scheduling (step, exponential, cosine, warmup, one-cycle) resolves that tension across a single training run.

Next: Backpropagation returns to the chain-rule mechanics from Calculus for DL, this time walking through a complete numerical backpropagation example end to end — the missing piece connecting how the gradients this whole category has been optimizing actually get computed.

Practice Question

Why does OneCycleLR need to know the total number of training steps in advance, unlike a schedule like ExponentialLR?

Related DL Notes

• Warmup Learning Rate
• Cosine Annealing
• AdamW