Model selection is the broader decision of which algorithm — and which hyperparameter settings for it — to actually use, combining algorithm choice and hyperparameter tuning into one coherent process rather than treating them separately.
The Full Model Selection Process
| Step | What Happens |
|---|---|
| 1 | Shortlist candidate algorithms based on the problem type and data characteristics |
| 2 | For each candidate, tune its hyperparameters via cross-validation |
| 3 | Compare each algorithm's best-tuned cross-validated performance |
| 4 | Select the overall best-performing algorithm + hyperparameter combination |
| 5 | Evaluate that final choice exactly once, on the untouched test set |
Comparing Multiple Algorithms Fairly
from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
candidates = {
"logreg": (LogisticRegression(max_iter=1000), {"C": [0.1, 1, 10]}),
"svm": (SVC(), {"C": [0.1, 1, 10], "gamma": [0.01, 0.1]}),
"random_forest": (RandomForestClassifier(random_state=42), {"n_estimators": [100, 200], "max_depth": [None, 10]}),
}
best_overall = None
for name, (model, param_grid) in candidates.items():
search = GridSearchCV(model, param_grid, cv=5, scoring="f1")
search.fit(X_train, y_train)
print(f"{name}: best CV F1 = {search.best_score_:.3f}, params = {search.best_params_}")
if best_overall is None or search.best_score_ > best_overall[1]:
best_overall = (name, search.best_score_, search.best_estimator_)
print("Selected model:", best_overall[0])Every candidate is compared using the same cross-validation scheme and metric — a fair, apples-to-apples comparison, avoiding the mistake of tuning one algorithm carefully while leaving another at its defaults.
Why Algorithm Choice and Hyperparameter Tuning Can't Be Fully Separated
An untuned SVM might genuinely underperform a well-tuned logistic regression, even if SVM would have won with proper tuning — comparing algorithms fairly requires tuning each one reasonably well first, not comparing default settings against each other.
Practical Use Cases
- Choosing between fundamentally different algorithm families for a new problem, not just tuning one already-chosen algorithm
- Documenting and justifying a model choice with a clear, reproducible comparison process
Common Mistakes
- Comparing algorithms at their default hyperparameters, unfairly favoring whichever one happens to have better defaults for this specific problem.
- Selecting the final model based on test-set performance across multiple candidates — this reuses the test set for selection, contaminating its role as an honest, final estimate.
Interview Relevance
Q: "How would you fairly compare Random Forest and SVM for a new classification problem?" Tune each algorithm's hyperparameters via the same cross-validation scheme and scoring metric, compare their best cross-validated scores, and only then evaluate the single winning combination once on the held-out test set — comparing untuned defaults would unfairly bias the comparison toward whichever algorithm happens to have better default settings.
Practice Question
You tune Random Forest carefully via grid search but only try SVM at its default settings, then conclude Random Forest is the better algorithm for your problem. What's wrong with this comparison?
