Likelihood flips the usual question about probability around: instead of asking "given fixed parameters, how probable is this data?", it asks "given this observed data, how probable does it make different candidate parameter values?" This reframing is exactly what training a model means.
Probability vs Likelihood — The Same Formula, a Different Question
| Probability | Likelihood | |
|---|---|---|
| What's fixed | Parameters \(\theta\) | The observed data |
| What varies | The data (hypothetically) | Candidate parameter values \(\theta\) |
| Question asked | "How likely is this outcome, given these parameters?" | "How well do these parameters explain the data I actually observed?" |
Numerical Example
You flip a coin 10 times and observe 7 heads. If the coin's true heads-probability is \(\theta\), the likelihood of observing exactly 7 heads out of 10 is:
Evaluating this for a few candidate values of \(\theta\): \(\mathcal{L}(0.5) \approx 0.117\), \(\mathcal{L}(0.7) \approx 0.267\), \(\mathcal{L}(0.9) \approx 0.057\). The data (7 heads out of 10) makes \(\theta=0.7\) noticeably more likely than \(\theta=0.5\) or \(\theta=0.9\) — this comparison across candidate parameter values is exactly what likelihood is for, and finding the \(\theta\) that maximizes it is the subject of the next note.
Code
from math import comb
def likelihood(theta, heads=7, flips=10):
return comb(flips, heads) * (theta ** heads) * ((1 - theta) ** (flips - heads))
for theta in [0.5, 0.7, 0.9]:
print(theta, likelihood(theta))
# 0.5 0.1171875
# 0.7 0.2668... -- highest among these three candidates
# 0.9 0.0574...Where This Shows Up in Deep Learning
Every standard loss function in deep learning can be derived from a likelihood: cross-entropy loss is (negative log) likelihood under a categorical distribution assumption; mean squared error is (negative log) likelihood under a Gaussian noise assumption. Understanding likelihood is what makes clear why these specific loss formulas were chosen, rather than treating them as arbitrary — the next note, Maximum Likelihood Estimation, makes this connection explicit and formal.
Common Mistakes
- Treating likelihood values as probabilities that should sum to 1 across parameter values — they don't; likelihood is only meaningful for comparing candidate parameters relative to each other, not as an absolute probability of the parameter being correct.
- Confusing "the parameters that best explain this specific observed data" with "the true underlying parameters" — likelihood gives you an estimate, not a certainty, especially with limited data.
Interview Relevance
Q: "What's the difference between probability and likelihood?" Probability answers "how likely is this outcome, given fixed known parameters?" Likelihood answers "how well do different candidate parameter values explain data I've already observed?" They use the same mathematical formula, but probability fixes the parameters and varies the outcome, while likelihood fixes the outcome (the observed data) and varies the parameters.
Practice Question
For the coin example above, would you expect \(\mathcal{L}(0.6)\) to be higher or lower than \(\mathcal{L}(0.7)\)? Compute it to check your intuition.
