Gradient descent is the algorithm that actually trains most ML models: starting from random parameters, repeatedly take a small step in the direction that reduces the loss the most — the negative gradient — until the loss stops improving.
Formula
\(\theta\) is a parameter being learned (e.g. a regression coefficient), \(J(\theta)\) is the cost/loss function, \(\frac{\partial J(\theta)}{\partial \theta}\) is its gradient with respect to \(\theta\), and \(\alpha\) (alpha) is the learning rate — how big a step to take. This update repeats until the loss converges.
Geometric Intuition — Rolling Downhill
Each step moves opposite the gradient — perpendicular to the contour line — shrinking steps as the surface flattens near the minimum.
Worked Numerical Example
Minimize \(f(x) = x^2\) (so \(\frac{df}{dx} = 2x\)), starting at \(x_0 = 10\), with learning rate \(\alpha = 0.1\):
| Step | x | gradient = 2x | x_new = x − α×gradient |
|---|---|---|---|
| 0 | 10 | 20 | 10 − 0.1×20 = 8.0 |
| 1 | 8.0 | 16 | 8.0 − 0.1×16 = 6.4 |
| 2 | 6.4 | 12.8 | 6.4 − 0.1×12.8 = 5.12 |
| 3 | 5.12 | 10.24 | 5.12 − 0.1×10.24 = 4.10 |
x = 10.0
alpha = 0.1
for step in range(10):
grad = 2 * x # derivative of x^2
x = x - alpha * grad
print(step, round(x, 4))
# Output shrinks toward 0.0 — the minimum of x^2Learning Rate — Why It's the Most Important Hyperparameter Here
| Learning Rate | What Happens |
|---|---|
| Too small (e.g. 0.001) | Converges, but takes many more steps than necessary — slow training |
| Good (e.g. 0.1 above) | Steadily shrinks toward the minimum in a reasonable number of steps |
| Too large (e.g. 1.1) | Overshoots the minimum and diverges — the loss gets worse each step |
Diverging example: same function, \(\alpha = 1.1\), \(x_0 = 10\): step 1 → \(x = 10 - 1.1(20) = -12\); step 2 → \(x = -12 - 1.1(-24) = 14.4\) — the value is growing in magnitude and flipping sign every step instead of shrinking toward zero.
Practical Use Cases
- Training linear regression and logistic regression when the closed-form solution isn't used
- Training every neural network — this exact update rule, applied to millions of parameters via backpropagation
Limitations
- Can get stuck in a poor local minimum for non-convex loss surfaces (common in deep learning, rare in classical linear models)
- Sensitive to feature scale — unscaled features distort the loss surface into a narrow valley, slowing convergence. See Feature Scaling.
Common Mistakes
- Picking a learning rate without ever checking the loss curve — always plot loss vs. iteration to catch divergence or painfully slow convergence early.
- Forgetting to scale features before gradient-based training — this alone often fixes slow or unstable convergence.
Interview Relevance
Q: "Your model's loss is oscillating wildly and increasing during training — what's the likely cause?" The learning rate is too high, causing gradient descent to overshoot the minimum on every step instead of converging toward it — the fix is to lower \(\alpha\).
Practice Question
Using \(f(x) = x^2\), \(x_0 = 5\), \(\alpha = 0.2\), manually compute \(x_1\), \(x_2\) and \(x_3\).
