Bayesian Statistics

By the end of this lesson, you will understand how Bayesian statistics updates prior beliefs with new evidence to produce a posterior probability, and you will implement a simple Beta-Binomial update in Python.

What it is

Bayesian statistics is a framework for updating the probability of a hypothesis as more evidence or information becomes available. Unlike frequentist statistics, which treats parameters as fixed but unknown constants, Bayesian statistics treats them as random variables with probability distributions.

The core mental model involves three components:

• Prior: The initial belief about a parameter before seeing data.
• Likelihood: The probability of observing the data given the parameter.
• Posterior: The updated belief after combining the prior and likelihood.

This relationship is formalized by Bayes' Theorem: P(Hypothesis | Data) = [P(Data | Hypothesis) * P(Hypothesis)] / P(Data).

Why it matters

• Small Data: It allows reasonable inference even when sample sizes are small by leveraging prior knowledge.
• Uncertainty Quantification: It provides full probability distributions for parameters, not just point estimates, giving a clearer picture of uncertainty.
• Sequential Updating: As new data arrives, the current posterior becomes the next prior, allowing continuous learning without reprocessing all historical data.
• Decision Making: It aligns naturally with business decisions where costs of errors vary, by optimizing expected utility rather than just p-values.

Syntax or steps

The simplest practical application uses conjugate priors, where the prior and posterior belong to the same distribution family. For binary outcomes (success/failure), we use the Beta distribution as the prior for the Binomial likelihood.

1. Define the prior parameters alpha (successes) and beta (failures).
2. Observe new data: k successes out of n trials.
3. Update the parameters: new_alpha = alpha + k and new_beta = beta + (n - k).
4. The resulting distribution represents the posterior belief.

Example

import numpy as np
from scipy.stats import beta

# 1. Define Prior Belief
# Assume we believe a coin is fair (50% heads). 
# A Beta(1,1) distribution is uniform, representing no strong prior bias.
prior_alpha = 1
prior_beta = 1

# 2. Observe Data
# We flip the coin 10 times and get 7 heads (successes) and 3 tails (failures).
observed_successes = 7
observed_failures = 3

# 3. Calculate Posterior Parameters
posterior_alpha = prior_alpha + observed_successes
posterior_beta = prior_beta + observed_failures

# 4. Analyze Results
# Mean of the posterior distribution
posterior_mean = posterior_alpha / (posterior_alpha + posterior_beta)

print(f"Prior Mean: {prior_alpha / (prior_alpha + prior_beta):.2f}")
print(f"Posterior Mean: {posterior_mean:.2f}")
print(f"95% Credible Interval: {beta.interval(0.95, posterior_alpha, posterior_beta)}")

Explanation: The code starts with a neutral prior (Beta(1,1)). After observing 7 heads and 3 tails, the posterior becomes Beta(8,4). The mean shifts from 0.50 to approximately 0.67, reflecting the evidence that the coin might be biased toward heads. The credible interval gives a range where the true probability likely falls.

Common mistakes

• Ignoring the Prior: Using an overly informative prior can skew results if the prior is wrong. Always check sensitivity by trying different priors.
• Confusing Probability with Likelihood: The likelihood is a function of the data given the parameter; the posterior is the probability of the parameter given the data.
• Forgetting Normalization: In complex models, calculating the denominator P(Data) is hard. MCMC methods often bypass this, but manual calculations must ensure probabilities sum to 1.
• Misinterpreting Credible Intervals: A 95% credible interval means there is a 95% probability the parameter lies within the range, unlike frequentist confidence intervals which refer to long-run coverage rates.

When to use it

Practice

Guided Exercise: Modify the example above to assume a prior belief that the coin is heavily biased toward tails (Beta(1, 10)). Observe 5 heads and 5 tails. What is the new posterior mean?

Challenge: Implement a sequential update. Start with Beta(1,1). Update with 3 heads, then print the mean. Then update with 2 tails, then print the mean again. Verify that doing it sequentially yields the same result as processing all 5 observations at once.

Quick check

Question: If your prior is Beta(2, 2) and you observe 4 successes and 1 failure, what are the posterior parameters?

Answer: Alpha = 2 + 4 = 6, Beta = 2 + 1 = 3. The posterior is Beta(6, 3).

Summary

Bayesian statistics provides a rigorous method for updating beliefs with evidence, transforming prior assumptions into posterior probabilities. By using conjugate priors like the Beta-Binomial model, analysts can easily quantify uncertainty and incorporate domain knowledge, making it ideal for scenarios with limited data or high-stakes decision-making.