Data Drift

Data drift is specifically a shift in the distribution of a model's input features over time — the model itself and the true feature-target relationship may be completely unchanged, but the data it now sees no longer resembles what it was trained on.

Data Drift vs Model Drift vs Concept Drift — Keeping the Terms Straight

Detecting Data Drift Without Any Labels

Because data drift only concerns input distributions, it can be monitored continuously without waiting for ground-truth outcomes — a genuine practical advantage over concept drift detection.

from scipy.stats import ks_2samp
import numpy as np

training_income = np.random.normal(50000, 15000, 1000)     # distribution at training time
production_income = np.random.normal(58000, 18000, 1000)    # distribution observed now

statistic, p_value = ks_2samp(training_income, production_income)
print(f"KS statistic: {statistic:.3f}, p-value: {p_value:.4f}")
# A small p-value (e.g. < 0.05) suggests the two distributions are
# statistically significantly different -- evidence of data drift

The Kolmogorov-Smirnov (KS) test is a standard statistical test for whether two samples come from the same distribution — a natural fit for comparing a feature's training-time distribution against its current production distribution, for continuous numeric features specifically.

Monitoring Every Feature, Systematically

import pandas as pd
from scipy.stats import ks_2samp

def check_all_features_for_drift(training_df, production_df, threshold=0.05):
    drift_report = []
    for column in training_df.select_dtypes(include="number").columns:
        stat, p_value = ks_2samp(training_df[column], production_df[column])
        drifted = p_value < threshold
        drift_report.append({"feature": column, "p_value": p_value, "drifted": drifted})
    return pd.DataFrame(drift_report).sort_values("p_value")

report = check_all_features_for_drift(training_data, production_data)
print(report[report["drifted"]])   # features flagged as significantly drifted

For Categorical Features — Chi-Squared Test

from scipy.stats import chi2_contingency
import pandas as pd

training_counts = training_df["city"].value_counts()
production_counts = production_df["city"].value_counts()

contingency_table = pd.DataFrame({"training": training_counts, "production": production_counts}).fillna(0)
chi2, p_value, dof, expected = chi2_contingency(contingency_table.T)
print(f"p-value: {p_value:.4f}")   # small p-value -> category proportions have shifted meaningfully

Why Data Drift Doesn't Always Mean the Model Is Now Wrong

A shift in input distribution doesn't automatically mean the learned feature-target relationship has broken — if the model generalizes well across the new range of values, performance may hold up fine despite the drift. This is exactly why data drift is an early warning signal worth investigating, not automatic proof that retraining is required — pair it with actual performance monitoring where possible.

Practical Use Cases

• Continuous, label-free monitoring that catches upstream data pipeline changes and genuine population shifts early
• Deciding when it's worth investing effort in fresh ground-truth evaluation before drift becomes a confirmed accuracy problem

Common Mistakes

• Treating every statistically significant drift signal as requiring immediate retraining, without checking whether it's actually degrading real performance.
• Only checking a handful of "obvious" features for drift instead of systematically checking all of them — drift can appear in unexpected places.

Interview Relevance

Q: "How would you detect data drift without waiting for ground-truth labels?" Statistically compare each feature's current production distribution against its training-time distribution — using the KS test for continuous features or a chi-squared test for categorical ones — flagging features with a significantly different distribution as a candidate early-warning signal, well before delayed labels would reveal an actual accuracy problem.

Practice Question

A KS test on a feature returns p-value=0.003. What does this suggest, and what would you check next before deciding whether to retrain?

Related ML Notes

• Model Drift
• ML Monitoring
• Hypothesis Testing