PyTorch Optimizers

A practical reference for PyTorch's built-in optimizers — every one already covered conceptually in the Optimization category, with exact syntax and typical default hyperparameters.

Common Optimizers, Quick Reference

Code — The Standard Training Step

import torch.optim as optim

optimizer = optim.AdamW(model.parameters(), lr=1e-4, weight_decay=0.01)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=50)

for epoch in range(50):
    for x_batch, y_batch in dataloader:
        optimizer.zero_grad()
        loss = loss_fn(model(x_batch), y_batch)
        loss.backward()
        optimizer.step()
    scheduler.step()

Per-Parameter-Group Learning Rates

# Different learning rates for different parts of a model -- e.g. for fine-tuning
optimizer = optim.Adam([
    {"params": model.backbone.parameters(), "lr": 1e-5},    # pretrained backbone: smaller LR
    {"params": model.head.parameters(), "lr": 1e-3}          # new head: larger LR
])

This is exactly the pattern from Fine-Tuning — passing a list of parameter groups, each with its own hyperparameters, instead of a single flat parameter list.

Common Mistakes

• Passing model.parameters() to the optimizer before freezing some layers with requires_grad=False — the optimizer captures the parameter list at construction time; frozen parameters with requires_grad=False simply won't be updated (since they have no gradient), but it's cleaner to filter them out explicitly.
• Creating a new optimizer instance every epoch instead of reusing one — this discards the optimizer's internal state (like Adam's momentum estimates), effectively resetting training progress each epoch.

Interview Relevance

Q: "How would you set up an optimizer to fine-tune a pretrained backbone with a small learning rate while training a new classification head with a larger one?" Pass a list of parameter group dictionaries to the optimizer constructor, each specifying its own "params" and "lr" — one group for the backbone's parameters with a small learning rate, another for the new head's parameters with a larger one, letting a single optimizer manage both with different update magnitudes.

Practice Question

Why would recreating the optimizer object fresh every epoch (instead of creating it once before the training loop) hurt Adam's effectiveness specifically?

Related DL Notes

• Adam Optimizer
• Learning Rate Scheduling
• PyTorch Training Loop