Practical guidance on choosing weight decay strength — building on the conceptual mechanism from Weight Decay and the AdamW-specific considerations from AdamW.
Typical Ranges in Practice
| Range | Typical Use |
|---|---|
| 0 (no weight decay) | Sometimes appropriate for very small models with limited overfitting risk, or when other regularization (dropout, data augmentation) already suffices |
| 1e-5 – 1e-4 | Light regularization, common as a general-purpose default for many deep networks |
| 1e-2 – 1e-1 | Heavier regularization, more common specifically with AdamW on large Transformer-based models, where this range has become a widely-used convention |
The Critical Detail: Use AdamW, Not Adam, for Meaningful Weight Decay
Recall directly from AdamW: plain Adam's weight_decay argument doesn't behave as true weight decay — it gets folded into the gradient before adaptive scaling, producing inconsistent, gradient-history-dependent regularization strength across different parameters. If weight decay is going to be tuned as a meaningful hyperparameter at all, using AdamW (not Adam) is a prerequisite for that tuning to behave predictably.
Code — A Weight Decay Sweep
import torch.optim as optim
results = {}
for wd in [0, 1e-5, 1e-4, 1e-3, 1e-2]:
model = build_model()
optimizer = optim.AdamW(model.parameters(), lr=1e-3, weight_decay=wd)
train(model, train_loader, optimizer, epochs=20)
val_acc = evaluate(model, val_loader)
results[wd] = val_acc
print(f"weight_decay={wd}: val_accuracy={val_acc:.4f}")Interaction With Learning Rate
Weight decay's effective strength interacts with the learning rate — since the update rule applies weight decay scaled by the learning rate (recall the AdamW formula from AdamW), changing the learning rate without reconsidering weight decay (or vice versa) can shift the effective regularization strength unintentionally, making these two hyperparameters worth tuning together rather than in complete isolation.
Common Mistakes
- Tuning weight decay using plain Adam rather than AdamW — as covered in AdamW, this produces inconsistent, hard-to-interpret regularization behavior that makes tuning results unreliable.
- Treating weight decay and dropout as entirely independent, unrelated hyperparameters to tune separately — both serve overlapping regularization purposes, and their combined effect (not just each one individually) is what ultimately matters for the overfitting/underfitting balance.
Interview Relevance
Q: "Why is it important to use AdamW rather than plain Adam when weight decay is a hyperparameter you intend to tune carefully?" Plain Adam's weight_decay is folded into the gradient before adaptive per-parameter scaling, meaning its effective regularization strength varies inconsistently depending on each parameter's own gradient history — breaking the clean, predictable relationship between the configured weight decay value and the actual regularization applied. AdamW decouples weight decay from this adaptive scaling, restoring a consistent, directly interpretable relationship that makes tuning it meaningful.
Practice Question
Why might weight decay and dropout need to be tuned together, rather than each in complete isolation from the other?
