This note formalizes gradient descent as an optimization algorithm in general terms — building on the intuitive introduction in Gradient Descent (Intro) — before the next three notes cover its three practical variants: batch, stochastic, and mini-batch.
The Optimization Problem, Stated Precisely
Training a neural network means searching for the weight vector \(\mathbf{w}^*\) that minimizes the loss function \(L\). Gradient descent solves this iteratively, not analytically — repeatedly taking small steps in the direction that locally decreases \(L\) the fastest:
Convex vs Non-Convex Loss Surfaces
| Surface Type | Property | Deep Learning Reality |
|---|---|---|
| Convex | Exactly one global minimum, no local minima or saddle points | True for simple models like linear/logistic regression |
| Non-convex | Many local minima, saddle points, flat regions | True for essentially every neural network's loss surface |
This matters practically: gradient descent on a neural network has no guarantee of finding the global minimum — only a local one, or a point where the gradient happens to be near zero (recall from Hessian that this could be a saddle point, not a true minimum). In practice, this turns out to matter less than the theory might suggest — empirically, most local minima found by gradient descent in large networks perform comparably well, and saddle points, not bad local minima, are the more common practical obstacle.
Iterative Descent, Visualized
Each step moves opposite to the gradient — as the surface flattens near the minimum, the gradient shrinks, so steps naturally get smaller.
Code — The General Algorithm
import torch
def gradient_descent(loss_fn, w_init, lr=0.1, steps=50):
w = w_init.clone().requires_grad_(True)
for step in range(steps):
loss = loss_fn(w)
loss.backward()
with torch.no_grad():
w -= lr * w.grad
w.grad.zero_()
return w
# Example: minimize f(w) = (w - 3)^2, whose minimum is clearly at w=3
f = lambda w: (w - 3) ** 2
w_final = gradient_descent(f, torch.tensor(0.0), lr=0.1, steps=50)
print(w_final) # converges toward 3.0Common Mistakes
- Assuming gradient descent always finds the global minimum for a neural network's non-convex loss — it only guarantees convergence to a stationary point (zero gradient), which could be a local minimum or a saddle point.
- Forgetting that "gradient descent" is a family of algorithms, not one fixed procedure — the next three notes cover meaningfully different variants based on how much data each update uses.
Interview Relevance
Q: "Does gradient descent guarantee finding the best possible neural network weights?" No — a neural network's loss surface is non-convex, so gradient descent can only guarantee convergence to a local minimum or a stationary point (which could even be a saddle point), not the global minimum. In practice, this is less catastrophic than it sounds — most local minima found in large, over-parameterized networks tend to generalize comparably well.
Practice Question
For \(f(w) = (w-5)^2 + 2\), what is the true minimum value and at what \(w\)? Starting from \(w=0\) with a small learning rate, would gradient descent eventually reach it?
