Bernoulli Naive Bayes handles purely binary features — a word either appears in a document or it doesn't, with no regard for how many times — a distinct modeling choice from Multinomial NB's frequency counts.
Formula
\(x_i \in \{0,1\}\) indicates whether feature \(i\) is present. \(p_{i,y}\) is the probability that feature \(i\) is present at all, among class-\(y\) training documents (regardless of how many times it occurs). Critically, this formula also explicitly penalizes the absence of a feature (via the \((1-p_{i,y})^{1-x_i}\) term) — something Multinomial NB doesn't directly account for.
Worked Example — Reusing the Naive Bayes Hub's Spam Data
This is exactly the example already computed in the Naive Bayes hub note — it's Bernoulli by construction, since "contains 'free'" and "contains 'meeting'" are pure yes/no features, with no frequency information involved:
| P(contains "free") | P(contains "meeting") | |
|---|---|---|
| Spam | 0.833 | 0.167 |
| Not Spam | 0.25 | 0.75 |
For a new email containing "free" but NOT "meeting," the calculation explicitly used \((1-0.167)\) and \((1-0.75)\) — the absence terms — which is exactly Bernoulli NB's formula in action, giving \(P(\text{spam}\mid x) \approx 0.943\).
from sklearn.naive_bayes import BernoulliNB
import numpy as np
# has_free, has_meeting -> spam(1)/not spam(0)
X_train = np.array([
[1,0],[1,0],[1,0],[1,0],[1,1],[0,0], # spam examples
[0,1],[0,1],[0,1],[1,0], # not spam examples
])
y_train = np.array([1,1,1,1,1,1, 0,0,0,0])
model = BernoulliNB()
model.fit(X_train, y_train)
print(model.predict_proba([[1, 0]])) # contains "free", not "meeting"Multinomial vs Bernoulli — The Key Difference
| Multinomial NB | Bernoulli NB | |
|---|---|---|
| Feature representation | Word counts (frequency) | Binary presence/absence |
| Accounts for absence explicitly? | Not directly | Yes — the \((1-p)^{1-x_i}\) term |
| "free" appearing 5 times vs 1 time | Treated differently (more evidence) | Treated identically (both just "present") |
| Best for | Longer documents where frequency carries signal | Shorter documents, or when presence alone is the meaningful signal |
Why Explicitly Modeling Absence Matters
Bernoulli NB's penalty for a feature's absence can be a genuine advantage: if "meeting" is common in ham emails, its absence in a new email is itself mild evidence against ham — information Multinomial NB simply doesn't use (a word count of 0 just contributes nothing, rather than actively counting as evidence).
Practical Use Cases
- Short-text classification where word presence, not frequency, is the meaningful signal
- Any dataset naturally represented as binary features (e.g. "has this attribute" checkboxes)
Common Mistakes
- Feeding raw word counts into
BernoulliNBwithout binarizing them first — scikit-learn does binarize automatically by default (binarize=0.0), but it's worth knowing this happens rather than assuming counts are used directly. - Choosing between Multinomial and Bernoulli NB arbitrarily instead of based on whether frequency genuinely carries signal for the task.
Interview Relevance
Q: "When would you choose Bernoulli Naive Bayes over Multinomial?" When feature presence/absence itself is the meaningful signal rather than frequency (e.g. short texts, or tasks where a word appearing once vs many times shouldn't be treated differently) — and when you specifically want the model to account for the evidential value of a feature's absence, which Multinomial NB doesn't directly capture.
Practice Question
Explain, using the formula, why Bernoulli NB treats an email containing "free" once identically to an email containing "free" ten times.
