Bayesian optimization takes hyperparameter search a step further than random search's blind sampling — using the results of previous trials to intelligently choose which hyperparameter combination to try next, converging on good regions of the search space more efficiently.
The Core Idea
Rather than sampling hyperparameter combinations independently at random, Bayesian optimization builds a probabilistic model — a surrogate model, commonly a Gaussian Process — of how validation performance likely relates to hyperparameter values, based on every trial run so far. It then uses this model to decide the next combination to try, explicitly balancing two competing goals:
| Goal | Meaning |
|---|---|
| Exploitation | Try combinations near where the surrogate model predicts strong performance |
| Exploration | Try combinations in regions the surrogate model is still highly uncertain about, since they might hide an even better result |
An "acquisition function" formalizes this tradeoff, scoring candidate next points by combining the surrogate model's predicted performance with its uncertainty, and the next trial is chosen to maximize this score.
Why This Is More Sample-Efficient Than Random Search
Random search treats every trial as completely independent — it learns nothing from previous results. Bayesian optimization explicitly uses accumulated knowledge from every previous trial to make progressively smarter choices about where to search next, typically needing meaningfully fewer total trials to find a comparably good (or better) hyperparameter combination — a genuinely valuable property given how expensive each individual deep learning training run can be.
Diagram
The surrogate model's uncertainty (shaded band) grows wider away from observed points — the next trial is chosen to balance predicted performance against this uncertainty.
Code — Using scikit-optimize
from skopt import gp_minimize
from skopt.space import Real, Integer
def objective(params):
lr, batch_size = params
model = build_model()
train(model, lr=lr, batch_size=int(batch_size))
val_acc = evaluate(model, val_loader)
return -val_acc # gp_minimize MINIMIZES, so negate accuracy to effectively maximize it
space = [Real(1e-5, 1e-1, "log-uniform"), Integer(16, 128)]
result = gp_minimize(objective, space, n_calls=20) # far fewer calls than exhaustive grid search
print(result.x, -result.fun)Common Mistakes
- Assuming Bayesian optimization guarantees finding the absolute global optimum — like random search, it's a heuristic search strategy, generally more sample-efficient but without an exhaustiveness guarantee the way grid search has (within its specified grid).
- Using Bayesian optimization for a search space with very few dimensions and cheap individual trials — its added complexity may not be worth it when random or grid search would already be fast and sufficient.
Interview Relevance
Q: "How does Bayesian optimization improve on random search's approach to hyperparameter tuning?" Random search samples every trial independently, learning nothing from previous results. Bayesian optimization builds a probabilistic surrogate model of how hyperparameters relate to performance based on all previous trials, and uses it to intelligently choose the next trial — balancing exploiting promising regions against exploring uncertain ones — typically requiring meaningfully fewer total trials to reach a comparable or better result, which matters significantly given how expensive individual deep learning training runs are.
Practice Question
Why does Bayesian optimization's acquisition function need to balance exploitation and exploration, rather than simply always choosing the point with the best predicted performance?
