Probability measures how likely an event is, on a scale from 0 (impossible) to 1 (certain). Every classifier you'll ever train outputs probabilities — not certainties — and this entire category builds the language for reasoning about that uncertainty precisely.
Core Definitions
| Term | Meaning |
|---|---|
| Experiment | Any process with an uncertain outcome (e.g. classifying one image) |
| Sample space (\(\Omega\)) | The set of all possible outcomes (e.g. {cat, dog, bird}) |
| Event | A subset of outcomes you care about (e.g. "the prediction is cat") |
| Probability, \(P(A)\) | A number in \([0,1]\) measuring how likely event \(A\) is |
The Two Axioms That Matter Most in Practice
The second axiom is exactly why a classifier's output layer uses softmax — it forces the predicted probabilities across all classes to sum to exactly 1, matching what a valid probability distribution over outcomes must satisfy.
Numerical Example
A 3-class image classifier outputs \(P(\text{cat})=0.7\), \(P(\text{dog})=0.2\), \(P(\text{bird})=0.1\). These satisfy both axioms: each is between 0 and 1, and they sum to exactly 1.0. The model isn't claiming certainty — it's claiming cat is 7 times more likely than bird, given this specific input.
Code
import torch
import torch.nn.functional as F
logits = torch.tensor([2.0, 0.5, -1.0]) # raw, unnormalized scores from a network
probs = F.softmax(logits, dim=0) # convert to a valid probability distribution
print(probs) # tensor([0.7275, 0.1624, 0.1101])
print(probs.sum()) # tensor(1.0000)Where This Shows Up in Deep Learning
Nearly every classification network's final layer produces a probability distribution over classes, and nearly every generative model (language models included) is, at its core, predicting a probability distribution over possible next outputs (the next word, the next pixel value). This category builds every concept — random variables, distributions, entropy, cross-entropy — that's needed to precisely define what a classifier's loss function is actually measuring.
Common Mistakes
- Treating a model's softmax output as a measure of certainty rather than of relative likelihood — a well-calibrated 0.7 probability should be correct about 70% of the time across many predictions, not "70% certain" in some looser sense.
- Forgetting probabilities must sum to 1 across all outcomes — using raw, un-normalized network outputs (logits) directly as if they were probabilities is a common beginner mistake.
Interview Relevance
Q: "Why does a classification network's output layer need softmax instead of just using the raw scores?" Raw scores (logits) can be any real number and don't sum to 1, so they can't be interpreted as probabilities. Softmax exponentiates and normalizes them so the output satisfies both probability axioms — non-negative, summing to exactly 1 — making the output a valid probability distribution over classes.
Practice Question
A binary classifier outputs \(P(\text{spam}) = 0.83\). What is \(P(\text{not spam})\), and which probability axiom did you use to find it?
