Conditional probability asks: given that we already know something happened, how does that change the probability of something else? It's the mathematical foundation of Bayes' theorem, and of how a classifier's prediction should be read in context.
Formula
Read as "the probability of \(A\) given \(B\)." It's the probability of both \(A\) and \(B\) happening, divided by the probability of \(B\) happening at all — restricting the sample space to only the cases where \(B\) is true.
Numerical Example
Among 100 emails: 30 are spam. Of those 30 spam emails, 27 contain the word "free." Of the 70 non-spam emails, 7 contain "free." What is \(P(\text{spam} \mid \text{contains "free"})\)?
Knowing the email contains "free" raises the probability of spam from a base rate of 30% to about 79.4% — this exact kind of reasoning, generalized and automated, is what a Naive Bayes spam classifier does.
Code
# Verifying the numerical example directly by counting
spam_and_free = 27
total_free = 27 + 7
p_spam_given_free = spam_and_free / total_free
print(p_spam_given_free) # 0.7941...Independence — A Special Case
Independence means knowing \(B\) happened gives you no new information about \(A\). Many deep learning assumptions (e.g. that training examples are independently and identically distributed, "i.i.d.") rely on this idea directly.
Where This Shows Up in Deep Learning
A classifier's softmax output, \(P(\text{class} \mid \text{input})\), is itself a conditional probability — the probability of each class, given the specific input observed. This framing (modeling \(P(y \mid x)\) rather than \(P(y)\) alone) is exactly what distinguishes a useful predictive model from just reporting the base rate of each class in the training data.
Common Mistakes
- Confusing \(P(A\mid B)\) with \(P(B\mid A)\) — these are generally very different numbers (this exact confusion is often called the "prosecutor's fallacy"), and correctly relating them requires Bayes' theorem, covered next.
- Assuming independence without justification — the i.i.d. assumption behind most ML training is a simplification, and violations of it (e.g. time-correlated data) can silently break standard training and evaluation procedures.
Interview Relevance
Q: "What does a classifier's softmax output actually represent, probabilistically?" It represents \(P(\text{class} \mid \text{input})\) — a conditional probability distribution over classes, given the specific input. This is different from \(P(\text{class})\) alone (the base rate in the training data), and different from \(P(\text{input}\mid\text{class})\), which a generative model would estimate instead.
Practice Question
In a population, 1% have a disease. A test is 95% accurate for people who have it (true positive rate) and 90% accurate for people who don't (true negative rate). Without computing the exact answer yet, explain qualitatively why \(P(\text{disease} \mid \text{positive test})\) might be surprisingly low.
