A/B Testing & Experimentation

By the end of this lesson, you will be able to design a statistically valid A/B test by defining clear hypotheses, ensuring proper randomization, and calculating sample size requirements.

What it is

A/B testing (or split testing) is a randomized experiment comparing two variants, A (control) and B (treatment), to determine which performs better on a specific metric. The core mental model is causal inference: by randomly assigning users to groups, you isolate the effect of the change from other variables. Key terms include null hypothesis (no difference exists), alternative hypothesis (a difference exists), p-value (probability of observing results if null is true), and statistical power (probability of detecting an effect if it exists).

Why it matters

• Data-driven decisions: Replaces intuition with evidence when changing UI, algorithms, or marketing copy.
• Risk mitigation: Prevents rolling out changes that might harm key metrics like conversion rate or retention.
• Optimization: Systematically improves user experience and business outcomes over time.
• Causal clarity: Distinguishes correlation from causation in complex systems.

Syntax or steps

1. Define Objective: Choose one primary metric (e.g., click-through rate). 2. Hypothesis: State expected direction and magnitude of impact. 3. Sample Size Calculation: Determine required participants based on baseline conversion, minimum detectable effect (MDE), significance level ($\alpha$), and power ($1-\beta$). 4. Randomization: Assign users to Control or Treatment using a consistent hash function. 5. Run & Monitor: Collect data until sample size is reached; avoid peeking at p-values mid-test. 6. Analyze: Use statistical tests (e.g., z-test for proportions) to evaluate significance.

Example

import numpy as np
from scipy import stats

# 1. Define parameters
baseline_conversion = 0.05  # Current CTR
mde = 0.01                  # Minimum Detectable Effect (absolute increase)
alpha = 0.05                # Significance level
power = 0.8                 # Statistical power

# 2. Calculate Sample Size per group (approximate formula for proportions)
# Using normal approximation for simplicity in this example
z_alpha = stats.norm.ppf(1 - alpha/2)
z_beta = stats.norm.ppf(power)
p_bar = baseline_conversion + mde/2

n_per_group = ((z_alpha * np.sqrt(2 * p_bar * (1 - p_bar)) + 
                z_beta * np.sqrt(baseline_conversion*(1-baseline_conversion) + 
                                 (baseline_conversion+mde)*(1-(baseline_conversion+mde))))**2) / (mde**2)

print(f"Required sample size per group: {int(np.ceil(n_per_group))}")

# 3. Simulate Test Results
np.random.seed(42)
n = int(np.ceil(n_per_group))
control_successes = np.random.binomial(n, baseline_conversion)
treatment_successes = np.random.binomial(n, baseline_conversion + mde)

# 4. Analyze using Z-test for two proportions
count = [control_successes, treatment_successes]
nobs = [n, n]
z_stat, p_value = stats.proportions_ztest(count, nobs)

print(f"Control Rate: {control_successes/n:.4f}")
print(f"Treatment Rate: {treatment_successes/n:.4f}")
print(f"P-value: {p_value:.4f}")
print("Significant?" , "Yes" if p_value < alpha else "No")

Explanation: The code first calculates how many users are needed to detect a 1% absolute lift with 80% confidence. It then simulates data generation and uses a standard z-test to compare the two proportions. If the p-value is less than 0.05, we reject the null hypothesis.

Common mistakes

• Peeking: Checking results before reaching the planned sample size inflates false positives. Fix: Pre-calculate duration/sample size and stick to it.
• Multiple Comparisons: Testing many metrics increases chance of finding a "significant" result by luck. Fix: Adjust significance levels (Bonferroni correction) or define one primary metric.
• Novelty Effects: Users may react differently initially. Fix: Run tests long enough to capture stable behavior (usually 1-2 weeks).
• Improper Randomization: Using session IDs instead of user IDs can lead to contamination. Fix: Ensure assignment is sticky per user.

When to use it

Method	Best For	Limitations
A/B Testing	Isolating impact of a single change on a specific metric.	Requires large traffic; slow for rare events.
Multivariate Testing	Testing combinations of elements simultaneously.	Exponentially larger sample sizes needed.
Observational Analysis	Exploring correlations when experiments aren't feasible.	Cannot prove causation due to confounders.

Practice

Guided Exercise: Modify the code above to calculate the sample size needed if the baseline conversion is 10% and you want to detect a relative lift of 5% (i.e., absolute MDE = 0.005). Hint: Update `baseline_conversion` to 0.10 and `mde` to 0.005. Note how the required sample size changes dramatically compared to the previous example.

Quick check

Question: Why is it dangerous to stop an A/B test early because the p-value looks significant? Answer: This is known as "peeking." Stopping early increases the Type I error rate (false positive) because the probability of seeing a significant result by chance accumulates over multiple looks at the data. You must pre-commit to a fixed sample size or use sequential testing methods designed for early stopping.

Summary

Valid A/B testing relies on rigorous experimental design: clear hypotheses, adequate sample sizes calculated beforehand, and strict adherence to analysis plans. By avoiding common pitfalls like peeking and improper randomization, analysts can confidently attribute performance changes to their interventions rather than noise.