Between feature extraction (nothing in the backbone trained) and full fine-tuning (everything trained) lies a whole spectrum of choices. This note gives a concrete decision framework for picking the right point on that spectrum.
The Full Spectrum
| Approach | What's Trained | Data Needed | Overfitting Risk | Performance Ceiling |
|---|---|---|---|---|
| Feature extraction | Only the new head | Least | Lowest | Limited by frozen features' fit to the new task |
| Partial fine-tuning | The head + a handful of later backbone layers | Moderate | Moderate | Higher — some backbone adaptation possible |
| Full fine-tuning | Every parameter in the model | Most | Highest | Highest — full backbone adaptation possible |
The Classic Decision Framework — A 2×2 Grid
| Target Task Similar to Pretraining | Target Task Different from Pretraining | |
|---|---|---|
| Small target dataset | Feature extraction — features already suit the task; little data means high overfitting risk with more trainable parameters | Partial fine-tuning — some adaptation is needed for the different domain, but limited data still constrains how much can be safely retrained |
| Large target dataset | Partial or full fine-tuning — enough data to safely adapt more of the model, likely improving on frozen features somewhat | Full fine-tuning — a genuinely different domain benefits from adapting the whole model, and enough data exists to do so safely |
This framework directly synthesizes the considerations from the previous three notes: dataset size determines overfitting risk from more trainable parameters; task similarity determines how much the pretrained features already suit the target task without adjustment.
Code — Implementing Each Point on the Spectrum
import torchvision.models as models
import torch.nn as nn
model = models.resnet50(weights="IMAGENET1K_V2")
model.fc = nn.Linear(model.fc.in_features, 10)
# --- Feature extraction ---
for name, param in model.named_parameters():
param.requires_grad = "fc" in name # only the new head trains
# --- Partial fine-tuning: unfreeze the last block too ---
for name, param in model.named_parameters():
param.requires_grad = "fc" in name or "layer4" in name
# --- Full fine-tuning ---
for param in model.parameters():
param.requires_grad = True # everything trains (typically with a small learning rate)A Practical Empirical Approach
Rather than committing to one point on this spectrum purely from theory, a common practical workflow starts with feature extraction (fast, cheap, low-risk baseline), then progressively tries partial and full fine-tuning if validation performance suggests the frozen features are the limiting factor — using held-out validation performance (see Validation Loop) to decide, empirically, whether the added risk and cost of more aggressive fine-tuning is actually paying off for the specific task and dataset at hand.
Common Mistakes
- Defaulting to full fine-tuning regardless of dataset size — with a genuinely small target dataset, this frequently overfits badly, performing worse than a simpler, more conservative feature-extraction baseline would have.
- Treating this decision as purely theoretical rather than validating it empirically — the 2×2 framework is a useful starting heuristic, but actual validation performance on your specific data is the ultimate deciding factor.
Interview Relevance
Q: "You have a small labeled dataset (a few hundred images) for a task quite different from ImageNet's natural photos (e.g. satellite imagery). What transfer learning approach would you start with?" Given the small dataset size, aggressive full fine-tuning risks significant overfitting despite the domain difference suggesting more adaptation would ideally help. A reasonable starting point is partial fine-tuning — unfreezing just the last few layers to allow some domain adaptation, while keeping most of the pretrained backbone frozen to limit overfitting risk — then validating empirically whether more or less unfreezing improves held-out performance.
Practice Question
Using the 2×2 framework, what approach would you recommend for a large (100,000+ examples), domain-similar target dataset?
