Explainable AI (XAI) is the set of techniques for understanding why a model made a specific prediction — increasingly essential as more accurate models (Random Forest, XGBoost, neural networks) have become harder to interpret directly than simple linear models.
Why This Matters Beyond Curiosity
| Reason | Concrete Example |
|---|---|
| Trust | A doctor won't act on a diagnosis model's output without understanding what drove it |
| Debugging | An unexpectedly dominant feature is often the first sign of data leakage |
| Regulation | Credit decisions often legally require a specific, explainable reason for denial |
| Fairness auditing | Checking whether a model relies inappropriately on a protected attribute or its proxy |
The Interpretability-Accuracy Tension
Simple models (linear/logistic regression, shallow decision trees) are inherently interpretable — you can read the reasoning directly off the coefficients or the tree structure. More flexible models (Random Forest, gradient boosting, neural networks) are typically more accurate but far harder to interpret directly — Explainable AI techniques exist specifically to bridge this gap, extracting understandable explanations from otherwise opaque models.
The Toolbox, At a Glance
| Technique | Scope | Full Note |
|---|---|---|
| Permutation Importance | Global | Which features matter most, overall |
| SHAP | Both global and local | Game-theoretic, per-prediction and aggregate explanations |
| LIME | Local | Explains one specific prediction via a simple local surrogate model |
See Model Interpretability for the important distinction between "interpretable" and "explainable," and Global vs Local Explanations for when each scope actually applies.
Minimal Working Example
from sklearn.inspection import permutation_importance
from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier(n_estimators=200, random_state=42).fit(X_train, y_train)
result = permutation_importance(model, X_val, y_val, n_repeats=10, random_state=42)
import pandas as pd
importances = pd.Series(result.importances_mean, index=X_val.columns).sort_values(ascending=False)
print(importances.head())Practical Use Cases
- Explaining individual high-stakes decisions (loan denial, medical risk score) to the person affected
- Debugging suspiciously strong model performance by checking which features are actually driving it
- Communicating model behavior to non-technical stakeholders and regulators
Common Mistakes
- Treating an explanation technique's output as proof of causation — every technique here explains what the model relied on, not necessarily true real-world cause and effect.
- Choosing a global technique when a local, per-prediction explanation was actually needed (or vice versa) — see Global vs Local Explanations.
Interview Relevance
Q: "Why has explainability become more important as ML models have gotten more accurate?" The most accurate modern models (ensembles, deep learning) are typically also the least directly interpretable — as their use spread into high-stakes, regulated domains (credit, healthcare, hiring), the need to explain individual decisions grew alongside the models' opacity, driving the development of post-hoc explanation techniques like SHAP and LIME.
Practice Question
A bank denies a loan using an XGBoost model. Regulation requires giving the applicant a specific reason. Which explainability technique would you reach for, and why?
