A/B Testing (ML)

Before fully replacing a production model, A/B testing compares a new candidate model against the current one on live traffic — providing real-world evidence of improvement, beyond what offline evaluation metrics alone can guarantee.

Why Offline Evaluation Alone Isn't Always Enough

A new model can show better metrics on a held-out offline test set, yet still underperform in actual production — the offline test set, however carefully constructed, is still a proxy for real-world conditions, and subtle differences (real user behavior, live data characteristics not fully captured offline) can occasionally cause offline improvements not to translate into real-world gains. A/B testing directly measures the new model's actual impact on live traffic, closing this gap.

The Core A/B Testing Setup

import random

def route_request(user_id, treatment_percentage=0.1):
    # Consistently route the SAME user to the SAME model version across
    # their session -- avoids inconsistent experience and confounded results
    if hash(user_id) % 100 < treatment_percentage * 100:
        return "model_b_new_candidate"
    else:
        return "model_a_current_production"

model_version = route_request(user_id)
prediction = models[model_version].predict(input_data)
log_for_analysis(user_id, model_version, prediction, outcome=None)   # outcome filled in later

Routing consistently by a hash of the user ID (rather than randomly per-request) ensures each individual user has a consistent experience throughout the test and keeps the statistical comparison clean — a user shouldn't bounce between model versions from one request to the next.

Statistical Significance — Not Jumping to Conclusions Early

A/B test results need to be evaluated for statistical significance (see Statistical Significance) before concluding one model genuinely outperforms the other — a difference observed early, or with a small sample, can easily be due to random chance rather than a real effect. Running the test for a sufficient duration and sample size, and applying a proper significance test, avoids prematurely rolling out a model based on noise.

Gradual Rollout — A Practical Risk Management Pattern

Common Mistakes

• Concluding a new model is better based on early results from a small sample or short test duration, without checking for statistical significance — this risks rolling out a change based on random noise rather than a genuine improvement.
• Routing individual requests randomly rather than consistently by user — this can expose the same user to inconsistent behavior across requests and complicate clean statistical comparison between the two groups.

Interview Relevance

Q: "Why would you A/B test a new model on live production traffic even after it already showed improved metrics on an offline test set?" An offline test set, however carefully constructed, is still a proxy for real-world production conditions — real user behavior and live data characteristics aren't always fully captured offline, and offline improvements don't always translate cleanly to real-world gains. A/B testing measures the new model's actual impact directly on live traffic, providing real-world evidence of improvement (or catching an unexpected regression) before committing to a full rollout, and does so with limited initial exposure through a gradual rollout strategy.

Practice Question

Why is routing the same user consistently to the same model version throughout an A/B test important, rather than randomizing per individual request?

Related DL Notes

• Statistical Significance
• Model Registry
• Model Drift