Bayesian Optimization

Bayesian optimization uses the results of previous hyperparameter trials to intelligently choose the next combination to try — rather than gridding or randomly sampling blindly, it builds a probabilistic model of "which regions of the search space look promising" and searches those more.

The Core Idea

Exploration vs Exploitation

This is the same fundamental tradeoff reinforcement learning agents face: exploitation means trying combinations near the best result found so far, likely to yield modest, reliable improvements; exploration means trying combinations in uncertain, untested regions, which might reveal a much better region entirely. A good acquisition function (like "expected improvement") balances both, rather than greedily exploiting the current best guess.

Python Implementation — Optuna

import optuna
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier

def objective(trial):
    n_estimators = trial.suggest_int("n_estimators", 50, 300)
    max_depth = trial.suggest_int("max_depth", 2, 20)
    min_samples_leaf = trial.suggest_int("min_samples_leaf", 1, 10)

    model = RandomForestClassifier(
        n_estimators=n_estimators, max_depth=max_depth,
        min_samples_leaf=min_samples_leaf, random_state=42,
    )
    scores = cross_val_score(model, X_train, y_train, cv=5, scoring="accuracy")
    return scores.mean()

study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=50)

print("Best params:", study.best_params)
print("Best score:", study.best_value)

Each call to objective() represents one trial — Optuna intelligently chooses the hyperparameters for each subsequent call based on all previous results, rather than sampling blindly.

Why This Is More Sample-Efficient

Grid and random search treat every trial as independent, ignoring what previous trials revealed. Bayesian optimization actively learns from the search history — after a few trials reveal that very small max_depth values consistently perform poorly, it stops wasting further trials in that region, focusing remaining budget on more promising areas instead.

Practical Use Cases

• Expensive-to-train models (deep neural networks, large boosting ensembles) where every trial is costly, and sample efficiency genuinely matters
• Larger search spaces where random search's blind sampling wastes too much budget on clearly poor regions

Advantages

• Typically finds good hyperparameters with fewer total trials than grid or random search
• Naturally adapts its search as more information becomes available

Limitations

• More complex to set up and understand than grid/random search
• The surrogate model itself has overhead — for very cheap-to-train models, this overhead can outweigh the sample-efficiency benefit
• Less trivially parallelizable than random search, since each trial ideally uses information from previous ones

Common Mistakes

• Reaching for Bayesian optimization on a cheap-to-train model with a small search space, where the setup overhead isn't worth it over simple random search.
• Running too few trials for the surrogate model to build a genuinely useful understanding of the search space.

Interview Relevance

Q: "Why is Bayesian optimization more sample-efficient than random search?" It uses a probabilistic model built from every previous trial's result to intelligently choose the next hyperparameter combination, focusing search effort on promising regions and away from clearly poor ones — random search, by contrast, samples blindly and independently, never learning from its own history within the search.

Practice Question

You're tuning a deep neural network where each training run takes 6 hours. Would you lean toward random search or Bayesian optimization, and why does the cost per trial matter for this decision?

Related ML Notes

• Random Search
• Grid Search
• Model Selection