This opening note of the Hyperparameter Tuning category gathers every hyperparameter already introduced throughout this hub into one reference, and previews the general progression of tuning strategies covered for the rest of this category.
Recap: Parameters vs Hyperparameters
Recall the precise distinction from Parameters vs Hyperparameters: parameters (weights, biases) are learned automatically via gradient descent; hyperparameters are chosen by you, before training, and never updated by the optimizer. This entire category is about choosing those hyperparameters well.
The Complete Catalog
| Hyperparameter | Controls | Covered In |
|---|---|---|
| Learning rate | Step size of every weight update | Learning Rate Tuning |
| Batch size | How many examples per gradient update | Batch Size Tuning |
| Epochs | How many full passes through the training data | Epochs Tuning |
| Network depth | Number of layers | Network Depth Tuning |
| Hidden units | Width of each layer | Hidden Units Tuning |
| Dropout rate | Regularization strength via random deactivation | Dropout Tuning |
| Weight decay | Regularization strength via weight shrinkage | Weight Decay Tuning |
| Optimizer choice | The specific update rule used | Optimizer Selection |
| Activation function | Non-linearity choice per layer | Activation Function Selection |
The Progression of Tuning Strategies, Previewed
The rest of this category moves through increasingly systematic approaches: manual, intuition-guided tuning (the individual hyperparameter notes that follow), then exhaustive Grid Search, then the often more efficient Random Search, then genuinely intelligent, sample-efficient Bayesian Optimization, and finally Optuna — a practical framework implementing these ideas directly.
General Tuning Philosophy
- Start with well-established defaults (covered per-hyperparameter in the following notes) rather than tuning everything from scratch.
- Tune the hyperparameters with the largest typical impact first — learning rate almost always matters most.
- Always tune against validation performance (see Validation Loop), never the test set — exactly the discipline from Dataset Train/Val/Test Split.
- Move to systematic/automated search once manual intuition is exhausted or a large hyperparameter space needs exploring efficiently.
Common Mistakes
- Tuning every hyperparameter simultaneously and exhaustively from the start — prioritizing the highest-impact hyperparameters (learning rate especially) first is a far more efficient use of limited compute and time.
- Tuning against the test set — this silently turns the test set into a second validation set, exactly the mistake flagged in Dataset Train/Val/Test Split.
Interview Relevance
Q: "If you could only tune one hyperparameter carefully due to limited time, which would you choose?" The learning rate — it's widely regarded as the single most impactful hyperparameter in deep learning, since an overly aggressive learning rate can prevent training from converging at all, while an overly conservative one wastes training time without necessarily reaching a better solution; most other hyperparameters have comparatively more forgiving, wider "good enough" ranges.
Practice Question
Why is it important to always tune hyperparameters against validation performance rather than test set performance?
