ML retraining is the process of updating a deployed model on fresh data — the direct response to model drift, and the step that closes the MLOps lifecycle's loop back into continuous training.
The Three Retraining Triggers
| Trigger | How It Works | Tradeoff |
|---|---|---|
| Scheduled | Retrain on a fixed calendar (weekly, monthly) | Simple, predictable — but may retrain unnecessarily, or too late for fast-moving drift |
| Drift-triggered | Retrain when data drift monitoring crosses a threshold | Responsive to actual change, but needs reliable drift detection in place first |
| Performance-triggered | Retrain when measured accuracy against ground truth drops below a threshold | Most directly tied to what matters, but limited by label delay |
In practice, mature systems often combine all three — a scheduled baseline cadence, with drift or performance alerts capable of triggering an earlier, unscheduled retrain.
A Basic Retraining Pipeline Pattern
def retraining_pipeline():
# 1. Pull the latest data
fresh_data = load_latest_training_data()
# 2. Validate it before training on it -- a corrupted upstream feed
# shouldn't silently train a broken model
if not validate_data_quality(fresh_data):
alert_team("Data validation failed -- retraining aborted")
return
# 3. Train a candidate model
candidate_model = train_model(fresh_data)
# 4. Evaluate the candidate against the SAME held-out test set used historically,
# for a fair, consistent comparison across retraining cycles
candidate_metrics = evaluate(candidate_model, fixed_test_set)
current_production_metrics = get_production_model_metrics()
# 5. Only promote if the candidate is genuinely better -- never blindly replace
if candidate_metrics["f1"] > current_production_metrics["f1"]:
register_and_promote(candidate_model)
alert_team(f"New model promoted: F1 improved from "
f"{current_production_metrics['f1']:.3f} to {candidate_metrics['f1']:.3f}")
else:
alert_team("Candidate model did not outperform production -- not promoted")
# Scheduled to run automatically, e.g. weekly, or triggered by a drift alert
retraining_pipeline()Why "Always Retrain, Always Promote" Is Dangerous
Automatically promoting every retrained model — without comparing it against the current production model on a consistent held-out set — risks silently deploying a worse model, especially if the fresh training data happens to be noisy or contains a pipeline bug. The evaluation-and-comparison gate in the pipeline above is what prevents an automated retraining system from becoming an automated regression-introduction system.
Retraining vs Fine-Tuning
| Full Retraining | Fine-Tuning / Incremental Update | |
|---|---|---|
| Approach | Train from scratch on the full (updated) dataset | Continue training an existing model on new data only |
| Cost | Higher — full training run | Lower — often faster, especially for large models |
| Risk | Lower — a completely fresh, consistent model | Higher — can drift away from original behavior in unintended ways ("catastrophic forgetting" for some model types) |
Practical Use Cases
- Any production model where the underlying data distribution is expected to shift over time — most real-world deployments
- Automating what would otherwise be a manual, easy-to-forget maintenance task
Common Mistakes
- Automatically promoting every retrained model without comparing it against the current production model first.
- Retraining on a fixed schedule alone with no drift or performance monitoring, missing fast-moving degradation between scheduled runs.
- Not validating fresh data quality before training on it, letting an upstream pipeline bug corrupt the next model generation.
Interview Relevance
Q: "Why shouldn't a retraining pipeline automatically promote every new model it produces?" Fresh training data can be noisy, incomplete, or affected by an upstream pipeline bug — automatically promoting without comparing the candidate against the current production model's performance on a consistent evaluation set risks silently deploying a regression instead of an improvement.
Practice Question
Design the trigger strategy for a fraud detection model where both fast-moving fraud pattern shifts and steady, gradual population change are both realistic concerns.
