A pretext task is the specific, auxiliary prediction problem a self-supervised model is actually trained on — chosen not because anyone cares about solving it for its own sake, but because solving it well forces the model to learn genuinely useful, general-purpose representations as a necessary byproduct.
Common Pretext Tasks, Across Domains
| Domain | Pretext Task | What It Forces the Model to Learn |
|---|---|---|
| NLP | Predict a masked word (MLM) or the next word | Grammar, semantics, world knowledge |
| Vision | Predict the rotation angle applied to an image | Object shape and orientation understanding |
| Vision | Colorize a grayscale image | Object identity and typical color associations |
| Vision | Predict the relative position of two image patches | Spatial and structural relationships within objects/scenes |
| Vision/NLP | Reconstruct a corrupted/masked input (autoencoding) | Compressed, information-dense representations |
Why the Choice of Pretext Task Matters So Much
A poorly chosen pretext task can be solved via a "shortcut" that doesn't require learning anything genuinely useful — for example, predicting relative patch position could theoretically be solved by detecting chromatic aberration artifacts near image boundaries, rather than learning real object structure, if the task design doesn't carefully account for this. A well-designed pretext task closes off such shortcuts, forcing the model to engage with the data's actual semantic content to solve it well.
The Two-Stage Usage Pattern
- Pretrain a model on the pretext task, using cheap, abundant, unlabeled data.
- Discard or repurpose the pretext task's specific output head; keep the learned encoder/backbone.
- Transfer that backbone to genuine downstream tasks — via feature extraction or fine-tuning, exactly the techniques from the Transfer Learning category.
Code
import torch
import torch.nn as nn
class RotationPretext(nn.Module):
"""A pretext task: predict which of 4 rotations (0, 90, 180, 270 degrees) was applied."""
def __init__(self, backbone, feature_dim):
super().__init__()
self.backbone = backbone # this is what we actually care about keeping
self.rotation_head = nn.Linear(feature_dim, 4) # DISCARDED after pretraining
def forward(self, rotated_image):
features = self.backbone(rotated_image)
return self.rotation_head(features)
# After pretraining: keep model.backbone, discard model.rotation_head entirely
# The backbone has learned useful visual features as a side effect of solving rotation predictionCommon Mistakes
- Evaluating a self-supervised model purely on how well it solves the pretext task itself — the pretext task's own accuracy is largely irrelevant; what matters is downstream task performance using the learned representations.
- Designing a pretext task with an exploitable shortcut that doesn't require genuine semantic understanding — this can produce a model that solves the pretext task perfectly while learning representations of little real downstream value.
Interview Relevance
Q: "Why doesn't it matter if a self-supervised model's pretext task accuracy is mediocre, as long as downstream task performance is good?" The pretext task is only a means to an end — it exists purely to force the model to learn useful, general representations as a byproduct of solving it. The actual measure of success is how well those learned representations transfer to genuine downstream tasks, not how well the model performs on the (often somewhat arbitrary) pretext task itself.
Practice Question
Why might "predict whether an image was flipped horizontally" be a weaker pretext task than "predict the rotation angle (0/90/180/270)" for learning useful visual representations?
