Error analysis is the practice of actually examining a model's mistakes individually, not just aggregate metrics — one of the most consistently valuable, and consistently underused, steps in a real deep learning project.
Why Aggregate Metrics Alone Aren't Enough
A model with 90% accuracy tells you almost nothing about which 10% of examples it gets wrong, or why — and that missing information is often exactly what's needed to know how to actually improve the model further. Error analysis fills this gap by directly examining specific failure cases.
The Core Error Analysis Process
- Collect a sample of the model's incorrect predictions on validation data.
- Manually review these examples, looking for systematic patterns — are errors concentrated in a specific class, a specific input characteristic, or a specific type of edge case?
- Categorize errors into buckets (e.g. "confuses class A and B specifically," "fails on low-quality/blurry images," "struggles with unusually short inputs").
- Prioritize fixing the buckets with the largest impact on overall performance, not necessarily the most conceptually interesting ones.
Code — Collecting and Reviewing Errors
import torch
model.eval()
errors = []
with torch.no_grad():
for x_batch, y_batch in val_loader:
predictions = model(x_batch).argmax(dim=1)
wrong_mask = predictions != y_batch
for i in wrong_mask.nonzero():
errors.append({
'input': x_batch[i],
'true_label': y_batch[i].item(),
'predicted_label': predictions[i].item(),
})
print(f"Total errors: {len(errors)}")
# Which true-class -> predicted-class confusions are most common?
from collections import Counter
confusion_pairs = Counter((e['true_label'], e['predicted_label']) for e in errors)
print(confusion_pairs.most_common(10))Using the Confusion Matrix as a Starting Map
The confusion matrix (covered fully in the Evaluation Metrics category) is often the natural first step into error analysis — it reveals which specific classes are most often confused with each other, directing manual review toward the specific, highest-impact error patterns rather than reviewing errors unguided.
Why This Directly Informs Next Steps
Systematic error patterns discovered here directly suggest specific fixes: a class confusion pattern might suggest more training examples for the confused classes, or a targeted architectural change; a failure mode tied to a specific input characteristic (blurry images, unusual lengths) might suggest targeted data augmentation (Data Augmentation Pipeline) or additional data collection for that specific case — much more actionable than a generic "try more hyperparameter tuning" response to a disappointing aggregate metric.
Common Mistakes
- Only ever looking at aggregate metrics, never individual failure cases — this misses systematic, fixable patterns that a single summary number can't reveal.
- Reviewing errors without any structured categorization — unstructured review tends to focus on whichever examples happen to be memorable or interesting, rather than systematically identifying the highest-impact error patterns.
Interview Relevance
Q: "A model achieves 92% accuracy, but the team wants to improve it further. What would you do before jumping to more hyperparameter tuning?" Conduct error analysis — directly examine a sample of the model's incorrect predictions, looking for systematic patterns (specific class confusions, specific input characteristics that consistently fail). This often reveals far more actionable, targeted next steps (more data for a specific weak case, a specific augmentation, a labeling error to fix) than blind hyperparameter tuning, which tends to produce only marginal gains once a model is already reasonably well-tuned.
Practice Question
Error analysis reveals that a model consistently misclassifies images taken in low-light conditions. What specific actions might this insight suggest, beyond generic hyperparameter tuning?
