Contrastive Learning

Contrastive learning is one of the most successful self-supervised approaches for computer vision — training a model to pull representations of "similar" (positive) pairs close together in embedding space, while pushing "dissimilar" (negative) pairs apart, closely generalizing the Contrastive/Triplet Loss ideas from the Loss Functions category.

The Core Recipe

1. Create a positive pair from a single example — commonly, two different random augmentations of the same image (crop, color jitter, blur).
2. Treat every other example in the batch as a negative relative to that positive pair.
3. Train the encoder so that positive pairs' embeddings are pulled close together, while negative pairs' embeddings are pushed apart.

The InfoNCE Loss

\(\mathbf{z}_i, \mathbf{z}_j\) are the positive pair's embeddings; the sum in the denominator runs over the positive pair plus every negative in the batch. \(\text{sim}\) is typically cosine similarity (a normalized dot product, see Dot Product); \(\tau\) is a temperature hyperparameter (structurally analogous to Temperature (Sampling)) controlling how sharply the loss penalizes near-miss negatives. This formula is, notably, exactly a softmax-based categorical cross-entropy (see Categorical Cross-Entropy) treating the correct positive pair as the "true class" among all pairs in the batch.

Why More Negatives Generally Helps

A larger pool of negative examples gives the model a richer, more discriminating training signal — it's easier to accidentally satisfy "be different from just a few negatives" than "be different from hundreds of diverse negatives simultaneously." This directly motivates the large-batch requirement of SimCLR and the memory-queue solution of MoCo, both covered next.

Code

import torch
import torch.nn.functional as F

def info_nce_loss(z_i, z_j, temperature=0.5):
    batch_size = z_i.shape[0]
    z = torch.cat([z_i, z_j], dim=0)                # 2N embeddings total
    z = F.normalize(z, dim=1)                          # cosine similarity via normalized dot products
    similarity = z @ z.T / temperature

    labels = torch.cat([torch.arange(batch_size) + batch_size, torch.arange(batch_size)])
    mask = torch.eye(2 * batch_size, dtype=torch.bool)
    similarity.masked_fill_(mask, float('-inf'))         # exclude self-similarity

    return F.cross_entropy(similarity, labels)   # exactly a softmax classification over all pairs

Common Mistakes

• Using augmentations that are too weak (barely changing the image) — this makes the positive-pair task trivially easy, providing little useful learning signal.
• Using augmentations so aggressive they destroy the image's actual semantic content — the two augmented views must still genuinely represent "the same thing," or the positive-pair assumption breaks down.

Interview Relevance

Q: "Why does contrastive learning typically benefit from a larger number of negative examples per training step?" More negatives give the model a richer, more discriminating training signal — distinguishing a positive pair from just a handful of negatives is a much easier, less informative task than distinguishing it from hundreds or thousands of diverse negatives simultaneously, which pushes the model to learn more precise, semantically meaningful representations.

Practice Question

Why is the InfoNCE loss described as being structurally identical to categorical cross-entropy, and what does the "correct class" correspond to in this context?

Related DL Notes

• Contrastive Loss
• SimCLR
• MoCo