Inference is what actually happens when a deployed model serves a real prediction — and getting the mechanics right, specifically avoiding a mismatch with how the model was trained, is where many production ML systems quietly go wrong.
Train-Serve Skew — The Central Risk
Train-serve skew happens when the preprocessing applied at inference time doesn't exactly match what was applied during training — even a small difference (a slightly different missing-value fill strategy, a feature computed with a subtly different formula) can silently degrade a deployed model's real-world accuracy far below what its offline evaluation suggested.
# DANGEROUS: reimplementing preprocessing logic separately for serving
def preprocess_for_inference(data):
# If this doesn't EXACTLY match the training-time preprocessing,
# you have train-serve skew -- and it can be very hard to detect
data["income_scaled"] = (data["income"] - 45000) / 15000 # hardcoded guesses at train-time stats!
return data
# SAFE: reuse the exact fitted pipeline from training, no reimplementation
pipeline = joblib.load("full_pipeline.pkl") # includes scaler, encoder, AND model
prediction = pipeline.predict(raw_new_data) # identical preprocessing, guaranteedThis is exactly why saving the full pipeline (not just the raw model) matters so much — it structurally eliminates train-serve skew, since the identical fitted transformations run at both training and inference time.
Loading the Model Once — Not Per-Request
# WRONG -- reloads the model from disk on every single request, adding real latency
def predict(request):
model = joblib.load("model.pkl") # slow disk I/O, every single call
return model.predict(request.features)
# RIGHT -- load once at application startup, reuse the in-memory object for every request
model = joblib.load("model.pkl") # happens ONCE
def predict(request):
return model.predict(request.features)For a large model, reloading from disk on every request can add tens or hundreds of milliseconds of unnecessary latency per prediction — a significant, entirely avoidable cost at any meaningful request volume.
Batch Inference for Efficiency
# If you need predictions for many rows at once, predict them together --
# far more efficient than looping and calling .predict() on one row at a time
predictions = model.predict(many_rows_at_once) # a single vectorized call
# AVOID this pattern for bulk prediction:
# predictions = [model.predict([row])[0] for row in many_rows] # much slowerHandling Prediction Failures Gracefully
def predict_safely(model, features):
try:
return model.predict(features)
except Exception as e:
# Log the failure, return a clear error -- never let a bad
# input silently crash the entire serving process
log_error(f"Prediction failed: {e}")
return {"error": "prediction_failed", "detail": str(e)}Practical Use Cases
- Every deployed model's actual serving logic, regardless of whether it's Flask, FastAPI, or a batch scoring script
- Diagnosing a deployed model's mysteriously worse real-world performance compared to its offline evaluation — train-serve skew is a common root cause
Common Mistakes
- Reimplementing preprocessing logic separately for the serving code instead of reusing the exact fitted pipeline from training.
- Loading the model fresh on every request instead of once at startup.
- Looping over individual rows for bulk prediction instead of a single vectorized batch call.
Interview Relevance
Q: "Your model performs well in offline evaluation but noticeably worse in production. What's a likely culprit?" Train-serve skew — check whether the production serving code's preprocessing exactly matches what was used during training and evaluation; a common cause is reimplementing preprocessing separately for the serving path instead of reusing the exact fitted pipeline, introducing subtle inconsistencies.
Practice Question
A team's serving code recomputes a feature's mean and standard deviation from each incoming request batch, instead of using the training-time fitted scaler. Explain why this is a train-serve skew bug.
