Optuna

This closing note of the Hyperparameter Tuning category covers Optuna — one of the most widely used practical frameworks for automated hyperparameter search, implementing Bayesian-optimization-style search alongside a genuinely useful additional feature: early termination of unpromising trials.

The Core Optuna Pattern

import optuna

def objective(trial):
    lr = trial.suggest_float('lr', 1e-5, 1e-1, log=True)
    batch_size = trial.suggest_categorical('batch_size', [16, 32, 64, 128])
    dropout = trial.suggest_float('dropout', 0.1, 0.6)
    num_layers = trial.suggest_int('num_layers', 2, 6)

    model = build_model(dropout=dropout, num_layers=num_layers)
    train(model, lr=lr, batch_size=batch_size)
    val_acc = evaluate(model, val_loader)
    return val_acc

study = optuna.create_study(direction='maximize')   # search for the HIGHEST validation accuracy
study.optimize(objective, n_trials=50)

print(study.best_params)
print(study.best_value)

Optuna's trial.suggest_* calls declaratively define the search space directly inside the objective function itself — internally, Optuna defaults to a Tree-structured Parzen Estimator (TPE), a specific, efficient Bayesian-optimization-style algorithm, to intelligently choose each new trial's hyperparameters based on all previous results.

Pruning — Stopping Bad Trials Early

def objective_with_pruning(trial):
    model = build_model(dropout=trial.suggest_float('dropout', 0.1, 0.6))
    optimizer = build_optimizer(trial.suggest_float('lr', 1e-5, 1e-1, log=True))

    for epoch in range(50):
        train_one_epoch(model, train_loader, optimizer)
        val_acc = evaluate(model, val_loader)

        trial.report(val_acc, epoch)   # report intermediate progress
        if trial.should_prune():        # Optuna decides if this trial looks unpromising
            raise optuna.TrialPruned()    # stop this trial EARLY, saving significant compute

    return val_acc

Pruning is a genuinely important practical advantage: rather than always running every trial to full completion (as random search and basic Bayesian optimization typically do), Optuna monitors a trial's progress partway through training and terminates it early if it's clearly performing far worse than other trials at the same point — saving substantial compute that would otherwise be wasted finishing a trial already known to be unpromising.

Visualizing Results

import optuna.visualization as vis

vis.plot_optimization_history(study).show()   # how the best value improved over trials
vis.plot_param_importances(study).show()        # which hyperparameters mattered most

The parameter importance plot directly connects back to the exact insight underlying random search's advantage from Random Search — most hyperparameter search problems have a small number of genuinely high-impact hyperparameters, and Optuna's importance analysis makes this concrete and visible for your specific search.

Common Mistakes

• Not using pruning for expensive training runs — this leaves significant compute savings unused, especially for tasks where a trial's poor trajectory is often visible well before its full training duration completes.
• Defining an overly wide or poorly-scaled search space (e.g. linear instead of log scale for learning rate) — this applies to Optuna exactly as it does to random search, wasting search budget on unlikely-to-be-useful regions.

Interview Relevance

Q: "What practical advantage does Optuna's pruning feature offer over standard Bayesian optimization or random search?" Pruning monitors a trial's intermediate progress during training and terminates it early if it's clearly underperforming relative to other trials at the same point — rather than always running every trial to full completion regardless of how it's trending. This can save substantial compute, particularly valuable given how expensive individual deep learning training runs typically are, letting the overall search budget be spent more efficiently on genuinely promising trials.

Key Takeaways — Hyperparameter Tuning

• Learning rate is typically the highest-impact hyperparameter, worth prioritizing and tuning systematically (e.g. via a learning rate range test) before others.
• Most individual hyperparameters (batch size, depth, width, dropout, weight decay) have well-established typical ranges and diagnostic symptoms — use validation loss/accuracy trends to guide adjustments.
• Grid search is exhaustive but scales combinatorially; random search is often more efficient for the same budget, since not every hyperparameter matters equally; Bayesian optimization (as implemented practically in Optuna) uses accumulated trial history to search even more efficiently, with pruning adding further compute savings.

Next: DL Project Development steps back from individual techniques to the complete end-to-end project lifecycle — problem definition through deployment and monitoring, applied to a full, realistic project.

Practice Question

Why might pruning be especially valuable when tuning hyperparameters for a large Transformer model, compared to a small, cheap-to-train MLP?

Related DL Notes

• Bayesian Optimization
• Random Search
• Experiment Tracking