This note zooms in on the gradient specifically as a vector object — its shape, how to interpret its individual components, and how it's represented in practice for a real neural network with millions of parameters.
Shape of the Gradient
For a loss function \(L(\mathbf{w})\) where \(\mathbf{w}\) is a weight vector of length \(n\), the gradient \(\nabla L(\mathbf{w})\) is also a vector of length \(n\) — one entry per weight, in the exact same shape as \(\mathbf{w}\) itself. This is by design: it lets you update every weight with one aligned vector subtraction, \(\mathbf{w} \leftarrow \mathbf{w} - \eta\nabla L\).
Reading Individual Components
| Component Value | Interpretation |
|---|---|
| Large positive | Increasing this weight would increase the loss a lot — decrease it during the update. |
| Large negative | Increasing this weight would decrease the loss a lot — increase it during the update. |
| Near zero | The loss is barely sensitive to this weight right now — it's near a local flat spot for this parameter. |
Numerical Example — A Weight Vector in Practice
Notice the third weight, with the largest-magnitude gradient component (2.4), moved the most — the update is proportional to how sensitive the loss is to each individual weight.
Code
import torch
w = torch.tensor([0.5, -0.2, 1.3], requires_grad=True)
loss = (w[0]-1)**2 + (w[1]+0.5)**2 + (w[2]-2)**2 # a toy loss over 3 weights
loss.backward()
print(w.grad) # the gradient vector, same shape as w: tensor([-1.0000, 0.6000, -1.4000])
learning_rate = 0.1
with torch.no_grad():
w -= learning_rate * w.grad # gradient descent update, applied to the whole vector at once
print(w)Where This Shows Up in Deep Learning
In practice, a network's gradient isn't one flat vector — PyTorch stores a separate gradient tensor for every parameter tensor in the model (each layer's weight matrix and bias vector gets its own .grad), matching that parameter's own shape exactly. Conceptually, though, they all together form one giant gradient vector over the network's full parameter space, and every optimizer (SGD, Adam, ...) operates on that structure.
Common Mistakes
- Forgetting that PyTorch accumulates gradients by default across multiple
.backward()calls — without callingoptimizer.zero_grad()before each new backward pass, gradients from previous steps silently add into the current one. - Interpreting the gradient vector's raw magnitude as "how wrong the model is" — the loss value itself measures that; the gradient measures sensitivity/direction, not error magnitude directly.
Interview Relevance
Q: "Why must PyTorch's optimizer.zero_grad() be called before every .backward()?" Because gradients accumulate (sum) into .grad by default across successive backward passes. Without resetting them to zero first, each new gradient computation would be added on top of the previous step's gradient, corrupting the update direction.
Practice Question
A weight vector has gradient \([-3, 0.5, -0.01]\). With a learning rate of 0.01, which weight will change the most in this update step, and in which direction (increase or decrease)?
