A parameter is learned automatically from data during training; a hyperparameter is set before training begins and controls how that learning happens — a distinction that trips up beginners constantly, but is simple once the examples click.
The Core Distinction
| Parameter | Hyperparameter | |
|---|---|---|
| Set by | The training algorithm, automatically | You, before training starts |
| Found via | Optimization (e.g. gradient descent) | Search (grid/random/Bayesian tuning) |
| Changes during training? | Yes — this IS what training does | No — fixed for the whole training run |
Examples Across Algorithms
| Algorithm | Parameters (learned) | Hyperparameters (you choose) |
|---|---|---|
| Linear Regression | Coefficients \(b_0, b_1, \dots\) | Regularization strength (if using Ridge/Lasso) |
| Logistic Regression | Weights \(w\), bias \(b\) | Regularization type and strength, solver |
| Decision Tree | Split thresholds, leaf values | max_depth, min_samples_leaf, criterion |
| Random Forest | Every individual tree's learned splits | n_estimators, max_features, tree hyperparameters |
| KNN | None — it's a lazy learner, nothing is "learned" during training | k, distance metric |
| SVM | Support vector weights | C, kernel type, gamma |
| Neural Network | All the connection weights | Learning rate, number of layers, layer sizes, batch size |
A Genuinely Instructive Edge Case — KNN
KNN has essentially no learned parameters at all — "training" just stores the data. Every meaningful setting (\(k\), the distance metric) is a hyperparameter you choose. This is exactly why KNN is called a "lazy" or "non-parametric" learner — see K-Nearest Neighbors.
Checking Both in scikit-learn
from sklearn.linear_model import LogisticRegression
model = LogisticRegression(C=1.0, penalty="l2") # C and penalty are hyperparameters, set before fitting
model.fit(X_train, y_train)
print(model.coef_, model.intercept_) # these ARE the learned parameters, only available AFTER fitting
print(model.get_params()) # the hyperparameters -- available even before fittingNotice: hyperparameters are visible via get_params() immediately, since they were set at construction; parameters like coef_ only exist after .fit() has actually run, since they're the output of training, not an input to it.
Practical Use Cases
- Correctly identifying what a specific "tuning" task is actually adjusting — hyperparameters, never parameters directly
- Understanding why hyperparameter search requires actually training multiple models, unlike parameter learning which happens within a single training run
Common Mistakes
- Referring to a model's learned coefficients as "hyperparameters" — a common, easy-to-catch terminology slip in interviews and technical writing.
- Assuming every algorithm has meaningful hyperparameters to tune — some (like plain linear regression via the Normal Equation, with no regularization) have essentially none.
Interview Relevance
Q: "Is the learning rate a parameter or a hyperparameter?" A hyperparameter — it's set before training begins and controls how the optimization proceeds (step size in gradient descent), but is never itself learned or adjusted by the training algorithm the way weights and coefficients are.
Practice Question
For a Random Forest, classify each as a parameter or hyperparameter: (a) the split threshold chosen at a specific node, (b) n_estimators, (c) the majority class predicted at a specific leaf.
