Few-Shot Learning

Few-shot learning means adapting a model to a new task or new classes from just a handful of examples — sometimes as few as one or two per class — a dramatic departure from the thousands or millions of examples standard supervised training assumes.

Two Distinct Ways Few-Shot Learning Happens

Prototypical Networks — A Classical Approach

For each class \(k\), compute a "prototype" \(\mathbf{c}_k\) — simply the average embedding of that class's few available examples, using a shared, pretrained embedding function \(f_\theta\). A new query example is then classified by finding which prototype it's closest to (via a distance metric, similar to Vector Norms) — no gradient-based fine-tuning is needed at all for the new classes themselves.

In-Context Learning — How Modern LLMs Do This

prompt = """
Classify the sentiment as positive or negative.

Review: "Amazing product, exceeded expectations!"
Sentiment: positive

Review: "Terrible quality, broke after one day."
Sentiment: negative

Review: "Works great, very happy with this purchase."
Sentiment:"""
# No weight updates happen at all -- the model infers the task purely from
# the two examples shown in the prompt (this is exactly "2-shot" in-context learning)

This is a genuinely remarkable emergent capability from What Is an LLM? — a sufficiently large pretrained model can infer a task's pattern purely from a few in-context demonstrations, with zero gradient updates, simply because next-token prediction at massive scale implicitly teaches this kind of pattern-completion behavior.

Code — A Simple Prototypical Network

import torch

def classify_few_shot(query_embedding, support_embeddings, support_labels, num_classes):
    prototypes = torch.stack([
        support_embeddings[support_labels == k].mean(dim=0) for k in range(num_classes)
    ])
    distances = torch.cdist(query_embedding.unsqueeze(0), prototypes)
    return distances.argmin(dim=1)   # classify by nearest prototype -- no gradient updates needed

Common Mistakes

• Confusing "few-shot" with "fine-tuning on a small dataset" — few-shot learning specifically means adapting with only a handful of examples, often without any gradient updates at all (in-context learning), a meaningfully different setting from standard fine-tuning on, say, hundreds of examples.
• Assuming in-context learning permanently changes a model — it doesn't; the model reverts to its original behavior the moment the prompt's examples are removed, since no weights were ever updated.

Interview Relevance

Q: "How does an LLM perform 'few-shot learning' without any gradient updates?" Through in-context learning — the model infers the task's pattern purely from a few example demonstrations placed directly in the prompt, using its pretrained next-token prediction capability to complete the pattern for a new query. No weights change; the model's parameters are entirely unaffected, and its adapted "behavior" exists only within that specific conversation/prompt context.

Practice Question

Why might prototypical networks struggle if the embedding function \(f_\theta\) wasn't already well-trained on data related to the new few-shot classes?

Related DL Notes

• What Is an LLM?
• Zero-Shot Learning
• Meta-Learning