The gradient collects every partial derivative of a function into a single vector — and for a neural network's loss, that vector points in the exact direction that increases the loss fastest. Gradient descent simply walks in the opposite direction.
Definition
For a function of \(n\) variables, the gradient \(\nabla f\) is a vector of \(n\) partial derivatives — one per input variable. For a neural network's loss \(L(w_1, \ldots, w_n)\), the gradient \(\nabla L\) is a vector with one entry per weight, telling you exactly how sensitive the loss is to each individual weight.
Numerical Example
Why the Gradient Points "Uphill"
The gradient always points in the direction of steepest increase; gradient descent moves in exactly the opposite direction, \(-\nabla f\), to decrease the loss fastest.
Code
import torch
x = torch.tensor(3.0, requires_grad=True)
y = torch.tensor(4.0, requires_grad=True)
f = x**2 + y**2
f.backward()
print(x.grad, y.grad) # tensor(6.) tensor(8.) -- matches [2x, 2y] at (3,4)Where This Shows Up in Deep Learning
Gradient descent's weight update rule is built directly on this: \(\mathbf{w} \leftarrow \mathbf{w} - \eta \nabla L(\mathbf{w})\), where \(\eta\) is the learning rate. Because \(\nabla L\) points toward increasing loss, subtracting it (moving in the \(-\nabla L\) direction) is exactly how every weight in the network gets nudged toward reducing the loss. This is covered in full in the Optimization category next.
Common Mistakes
- Forgetting the minus sign — the gradient points uphill; forgetting to negate it in a weight update would make the loss get worse, not better.
- Confusing the gradient (a vector) with a single derivative (a scalar) — the gradient only makes sense for functions of more than one variable, which is every practical loss function in deep learning.
Interview Relevance
Q: "Why does gradient descent subtract the gradient instead of adding it?" The gradient points in the direction of steepest increase of the function. Since training wants to minimize the loss, weights are updated in the opposite direction — hence subtracting \(\eta\nabla L\), not adding it.
Practice Question
For \(f(x,y) = 3x^2 + 5y\), compute the gradient \(\nabla f\) and evaluate it at \((x,y) = (2, -1)\). In which direction would gradient descent move from this point?
