A pretrained model is one already trained on a large, general dataset, made available for others to reuse — the actual starting point every transfer learning workflow begins from.
Where Pretrained Models Come From
| Domain | Common Pretraining Dataset | Common Pretrained Models |
|---|---|---|
| Computer vision | ImageNet (1.2M+ labeled images, 1000 classes) | ResNet, EfficientNet, ConvNeXt (see the CNN Architectures category) |
| NLP | Massive text corpora (Wikipedia, books, web text) | BERT, RoBERTa, GPT-family models |
| Multimodal | Large paired image-text datasets | CLIP and related vision-language models |
Why Using a Pretrained Model Saves So Much
- Compute: the original pretraining (potentially thousands of GPU-hours) never needs to be repeated.
- Data: your target task can use a far smaller labeled dataset than would be needed training from scratch, since general knowledge is already captured.
- Time: adapting a pretrained model to a new task typically takes hours, not the days/weeks full pretraining requires.
Code — Loading Pretrained Models in Practice
import torchvision.models as models
from transformers import AutoModel
# Vision: many architectures available with pretrained ImageNet weights
resnet = models.resnet50(weights="IMAGENET1K_V2")
efficientnet = models.efficientnet_b0(weights="IMAGENET1K_V1")
# NLP: the Hugging Face Hub hosts thousands of pretrained models
bert_model = AutoModel.from_pretrained("bert-base-uncased")
roberta_model = AutoModel.from_pretrained("roberta-base")Choosing a Pretrained Model — What Matters
| Consideration | Why It Matters |
|---|---|
| What it was originally trained on | Closer alignment with your target domain generally transfers better |
| Model size | Larger pretrained models often have richer general knowledge, but cost more to run and fine-tune |
| License and usage terms | Not every pretrained model is freely usable for every purpose — worth checking explicitly before deploying |
Common Mistakes
- Defaulting to the largest available pretrained model without considering compute constraints — a smaller model that's easier to fine-tune and deploy is often the more practical choice, especially early in a project.
- Assuming pretrained model weights are always free to use commercially without checking licensing — usage terms genuinely vary across different pretrained models and providers.
Interview Relevance
Q: "What factors would you weigh when choosing which pretrained model to start from for a new project?" How closely the pretraining data/task aligns with your target domain (closer alignment generally transfers better), the model's size relative to your compute and latency constraints, and licensing/usage terms for your intended application — not just raw benchmark performance in isolation.
Practice Question
For a medical image classification task with a small labeled dataset, would you expect a general ImageNet-pretrained model to transfer well? What might limit how well it transfers?
