Feature extraction is the simplest, cheapest form of transfer learning: keep the entire pretrained model completely frozen, and train only a small new "head" (typically a single linear layer) on top of its outputs.
The Setup
The pretrained backbone acts purely as a fixed feature extractor — it processes the input exactly as it always would, producing rich, general representations, but none of its own weights ever get updated. Only the new head's (typically few) parameters are trained on your target task's data.
Why This Approach Is Attractive
| Advantage | Why |
|---|---|
| Extremely fast to train | Only a small head's worth of parameters needs gradient computation and updating |
| Works well with very little data | Far fewer trainable parameters means far less risk of overfitting on a small target dataset |
| Preserves the pretrained knowledge exactly | Zero risk of "forgetting" anything learned during pretraining, since nothing in the backbone changes |
When Feature Extraction Works Best
This approach works particularly well when the target task is reasonably similar to what the model was originally pretrained on — the pretrained features are already well-suited, and only a lightweight mapping from those features to your specific output categories needs to be learned. When the target task is meaningfully different from the pretraining domain, feature extraction alone may not adapt well enough, motivating the fine-tuning approaches covered in the next two notes.
Code
import torch
import torch.nn as nn
import torchvision.models as models
backbone = models.resnet50(weights="IMAGENET1K_V2")
# Freeze EVERY parameter in the pretrained backbone
for param in backbone.parameters():
param.requires_grad = False
# Replace the final layer with a new, TRAINABLE head for your specific task
backbone.fc = nn.Linear(backbone.fc.in_features, 10) # this new layer's params DO require grad by default
# Only the new head's parameters will actually receive gradient updates during training
optimizer = torch.optim.Adam(backbone.fc.parameters(), lr=0.001) # note: only .fc.parameters(), not the whole modelCommon Mistakes
- Forgetting to explicitly set
requires_grad = Falseon the frozen backbone's parameters — without this, the backbone's weights would still receive gradients and be updated during training, defeating the entire point of feature extraction. - Passing the whole model's parameters (including the frozen backbone) to the optimizer — even with
requires_grad=Falsecorrectly set, it's cleaner and clearer to explicitly optimize only the new head's parameters.
Interview Relevance
Q: "When would feature extraction (fully frozen backbone) be preferable to fine-tuning any of the backbone's own weights?" When the target dataset is small (reducing overfitting risk from having fewer trainable parameters matters a lot), when the target task is reasonably similar to the original pretraining task (so the pretrained features are already well-suited without adjustment), or when training speed/compute is a significant constraint, since only the small new head needs gradient computation.
Practice Question
Why does feature extraction carry a much lower overfitting risk than full fine-tuning, when working with a very small target dataset?
