Step Decay

Step decay is the simplest and most widely understood learning rate schedule: keep the learning rate constant for a fixed number of epochs, then drop it suddenly by a fixed factor, and repeat.

Formula

\(s\) is the step size (how many epochs between drops), \(\gamma\) is the decay factor (e.g. 0.5 to halve, or 0.1 for a 10x drop), and \(\lfloor t/s\rfloor\) counts how many complete "steps" of \(s\) epochs have passed.

Numerical Example

With \(\eta_0=0.1\), \(s=10\), \(\gamma=0.5\): epochs 0–9 use \(\eta=0.1\); epochs 10–19 use \(\eta=0.1\times0.5=0.05\); epochs 20–29 use \(\eta=0.1\times0.25=0.025\); and so on — a clean halving every 10 epochs.

Graph

Code

import torch.optim as optim
from torch.optim.lr_scheduler import StepLR

optimizer = optim.SGD(model.parameters(), lr=0.1)
scheduler = StepLR(optimizer, step_size=10, gamma=0.5)

for epoch in range(30):
    # ... training loop for this epoch ...
    scheduler.step()
    print(scheduler.get_last_lr())   # [0.1]*10, then [0.05]*10, then [0.025]*10

Common Mistakes

• Choosing a step size and decay factor without watching validation loss — the "right" schedule is data-dependent; a common practical approach is to drop the learning rate specifically when validation loss plateaus, rather than at a rigid, pre-fixed schedule.
• Using too aggressive a decay factor (e.g. \(\gamma=0.01\)) — this can effectively freeze learning almost entirely after the first drop, wasting the remaining training budget.

Interview Relevance

Q: "What's a potential downside of step decay's sudden drops compared to a smoother schedule like cosine annealing?" The abrupt drop can cause a brief, noticeable jump in the loss curve right at the drop point, since the optimizer's step size suddenly changes rather than adapting gradually — smoother schedules avoid this discontinuity, though step decay's simplicity and interpretability remain genuine advantages.

Practice Question

With \(\eta_0=0.2\), \(s=5\), \(\gamma=0.1\), what is the learning rate at epoch 12?

Related DL Notes

• Learning Rate Scheduling
• Exponential Decay
• Cosine Annealing