Random search replaces grid search's exhaustive combination testing with randomly sampled hyperparameter combinations — and, perhaps counterintuitively, often finds better results with the same compute budget.
The Core Idea
import random
def sample_hyperparameters():
return {
'lr': 10 ** random.uniform(-5, -1), # sampled on a LOG scale -- appropriate for learning rate
'batch_size': random.choice([16, 32, 64, 128]),
'dropout': random.uniform(0.1, 0.6)
}
best_config, best_val_acc = None, 0
num_trials = 20 # far fewer than a full grid, yet often finds comparable or better results
for _ in range(num_trials):
config = sample_hyperparameters()
model = build_model(dropout=config['dropout'])
train(model, lr=config['lr'], batch_size=config['batch_size'])
val_acc = evaluate(model, val_loader)
if val_acc > best_val_acc:
best_val_acc, best_config = val_acc, configWhy Random Search Often Outperforms Grid Search at the Same Cost
A well-known empirical finding (Bergstra & Bengio, 2012): in practice, not every hyperparameter matters equally — a few tend to have a much larger impact on final performance than the rest. Grid search wastes a large fraction of its budget exhaustively varying every hyperparameter combination, including many along the (typically several) low-impact dimensions. Random search, by sampling every hyperparameter independently on every trial, naturally explores the important dimensions more thoroughly per unit of compute — for the same total number of trials, it tends to cover the truly consequential hyperparameters' ranges more densely.
Diagram — Why This Matters
With the same total trial budget, random search's scattered points give each individual hyperparameter dimension broader, less redundant coverage than a grid's rigid, repetitive spacing.
Common Mistakes
- Sampling a hyperparameter like learning rate uniformly on a linear scale instead of a logarithmic one — learning rate's meaningful effect spans several orders of magnitude, and linear sampling wastes most trials in a narrow, less-informative range.
- Using too few trials relative to the number of hyperparameters being tuned — random search's advantage relies on having enough trials to meaningfully cover the important dimensions; too few trials can still miss good combinations by chance.
Interview Relevance
Q: "Why can random search often outperform grid search for the same computational budget?" Not every hyperparameter has equal impact on final performance — typically a small number matter much more than the rest. Grid search's exhaustive combinations waste significant budget varying every dimension equally, including low-impact ones. Random search, sampling each hyperparameter independently every trial, naturally provides denser coverage of the high-impact dimensions for the same total trial count, since it doesn't waste structure enforcing uniform coverage across every dimension including unimportant ones.
Practice Question
Why is it important to sample a hyperparameter like learning rate on a logarithmic scale rather than a linear scale during random search?
