Fine-tuning goes a step further than feature extraction: continue training some or all of the pretrained model's own weights on your target task's data — adapting the pretrained knowledge itself, not just adding a new layer on top of it unchanged.
The Key Difference From Feature Extraction
| Feature Extraction | Fine-Tuning | |
|---|---|---|
| Backbone weights | Completely frozen, never updated | Actively updated via gradient descent, adapting to the new task |
| Risk | Minimal — pretrained knowledge fully preserved | Real risk of "catastrophic forgetting" — losing useful pretrained knowledge if updated too aggressively |
| Potential performance ceiling | Limited by how well the frozen features already suit the new task | Higher — the model can genuinely adapt its internal representations to the new task |
The Critical Practical Detail: A Much Smaller Learning Rate
Fine-tuning almost always uses a substantially smaller learning rate than training from scratch would — commonly 10x to 100x smaller. The pretrained weights already encode valuable, carefully-learned knowledge; a large learning rate risks destroying that knowledge with large, disruptive updates before the model has a chance to gently adapt it toward the new task. This directly echoes the Learning Rate discussion of fine-tuning needing much smaller rates than training from scratch.
Code
import torch
import torch.nn as nn
import torchvision.models as models
model = models.resnet50(weights="IMAGENET1K_V2")
model.fc = nn.Linear(model.fc.in_features, 10) # new head for the target task
# Fine-tuning: ALL parameters remain trainable (requires_grad=True by default)
# but with a MUCH smaller learning rate than typical from-scratch training
optimizer = torch.optim.Adam(model.parameters(), lr=1e-5) # note: 1e-5, not a typical 1e-3
# Common practice: use an even smaller learning rate for the pretrained backbone
# than for the freshly-initialized new head, since the head needs to learn faster
optimizer = torch.optim.Adam([
{"params": model.fc.parameters(), "lr": 1e-3}, # new head: normal learning rate
{"params": [p for n, p in model.named_parameters() if "fc" not in n], "lr": 1e-5} # backbone: much smaller
])Catastrophic Forgetting — The Real Risk
If fine-tuning is too aggressive (learning rate too high, too many epochs, or too little target data relative to the model's capacity), the model's weights can drift far enough from their pretrained values that the general knowledge captured during pretraining gets significantly degraded or lost — sometimes leaving the model worse off than a more conservative feature-extraction approach would have. This tension — how much to let the model adapt versus how much pretrained knowledge to preserve — is exactly what the next note's spectrum of approaches addresses directly.
Common Mistakes
- Using the same learning rate for fine-tuning as you would for training from scratch — this is one of the most common causes of catastrophic forgetting and degraded fine-tuning results.
- Fine-tuning on a very small target dataset without any regularization — with limited data and a large number of trainable parameters, fine-tuning can overfit rapidly, especially on the last few percent of training.
Interview Relevance
Q: "Why does fine-tuning typically use a much smaller learning rate than training a model from scratch?" The pretrained weights already encode valuable, carefully-learned representations from a large amount of prior training. A large learning rate risks applying large, disruptive updates that can quickly degrade or overwrite this useful knowledge — a failure mode called catastrophic forgetting — before the model has a chance to gently adapt its existing knowledge to the new task. A small learning rate allows careful, incremental adaptation instead.
Practice Question
Why might using different learning rates for the new head versus the pretrained backbone (as shown in the code example) make sense?
