This note covers freezing layers as a mechanic in its own right — the precise, practical control over exactly which parts of a pretrained model get updated during training, and the common strategies for deciding what to freeze.
The Mechanic, Precisely
In PyTorch, every parameter has a requires_grad flag. Setting it to False stops that parameter from accumulating gradients during .backward() and stops the optimizer from ever updating it — this is exactly what "freezing" means at the implementation level.
Common Freezing Strategies
| Strategy | What's Frozen | What's Trained |
|---|---|---|
| Full freeze (feature extraction) | The entire pretrained backbone | Only the new task-specific head |
| Freeze early layers only | The first several layers (most general knowledge) | Later layers + the new head |
| Progressive unfreezing | Everything, initially — then gradually unfrozen, layer by layer, over training | Starts with just the head, expanding outward over time |
Why Freeze Only Early Layers, Specifically
Recalling the general-to-specific pattern from What Is Transfer Learning? — early layers capture broadly useful, general features, while later layers capture more task-specific patterns. A common middle-ground strategy freezes only the early layers (preserving their general knowledge exactly) while allowing later layers to adapt more freely to the new task's specifics.
Code — Freezing Specific Layers
import torchvision.models as models
model = models.resnet50(weights="IMAGENET1K_V2")
# Freeze only the early layers; leave later layers and the head trainable
layers_to_freeze = [model.conv1, model.bn1, model.layer1, model.layer2]
for layer in layers_to_freeze:
for param in layer.parameters():
param.requires_grad = False
# model.layer3, model.layer4, and model.fc remain trainable
trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
total_params = sum(p.numel() for p in model.parameters())
print(f"Training {trainable_params:,} of {total_params:,} total parameters")Code — Progressive Unfreezing
# Start with everything frozen except the new head
for param in model.parameters():
param.requires_grad = False
model.fc.requires_grad_(True)
# ... train for a few epochs with just the head trainable ...
# Then progressively unfreeze earlier layers, epoch by epoch
for param in model.layer4.parameters():
param.requires_grad = True
# ... continue training, now allowing layer4 to adapt too ...Common Mistakes
- Freezing layers but still passing their parameters to the optimizer with a non-zero learning rate expecting them to stay unchanged based on
requires_gradalone — this is technically correct (frozen params won't accumulate gradients, so nothing happens), but it's cleaner and less error-prone to only pass genuinely trainable parameters to the optimizer. - Unfreezing too many layers too early in progressive unfreezing — this risks disrupting well-learned pretrained weights before the new head has stabilized enough to provide a sensible training signal.
Interview Relevance
Q: "What does 'freezing' a layer actually mean at the implementation level?" Setting that layer's parameters' requires_grad flag to False, which stops PyTorch's autograd from computing or accumulating gradients for those parameters during the backward pass, and consequently stops the optimizer from ever updating them — the layer's weights remain exactly as they were when frozen, throughout all subsequent training.
Practice Question
Why might progressive unfreezing (gradually unfreezing layers over the course of training) be safer than unfreezing the entire model all at once from the very first training step?
