Once the backward pass has computed each layer's error signal \(\boldsymbol\delta^{(l)}\) (see Backward Pass), getting the actual gradients for that layer's weights and biases is a short, direct final step.
Formula
The weight gradient is the outer product of the layer's error signal and its cached input (from Forward Pass) — this is precisely why that input had to be cached. The bias gradient is simply the error signal itself, since the bias contributes directly and equally to every part of \(\mathbf{z}^{(l)}\) without being scaled by any input.
Why an Outer Product
Each entry \(\frac{\partial L}{\partial W_{ij}}\) measures how sensitive the loss is to one specific weight connecting input neuron \(j\) to output neuron \(i\). Intuitively, this sensitivity should scale with both: how much error is currently flowing out of neuron \(i\) (\(\delta_i\)), and how strongly input \(j\) was active when this weight was used (\(a_j\)). The outer product \(\boldsymbol\delta^{(l)}(\mathbf{a}^{(l-1)})^\top\) produces exactly this pairwise product for every weight in the matrix simultaneously.
Numerical Example
\(\boldsymbol\delta^{(l)} = [0.2, -0.1]\) (2 output neurons), \(\mathbf{a}^{(l-1)} = [1.0, 0.5, 2.0]\) (3 input neurons):
The result has shape \((2,3)\) — exactly matching \(\mathbf{W}^{(l)}\)'s own shape, as it must, since this is the gradient for every individual weight in that matrix.
Code
import torch
delta = torch.tensor([0.2, -0.1])
a_prev = torch.tensor([1.0, 0.5, 2.0])
grad_W = torch.outer(delta, a_prev) # exactly the outer product formula
grad_b = delta.clone()
print(grad_W)
# tensor([[ 0.2000, 0.1000, 0.4000],
# [-0.1000, -0.0500, -0.2000]])
print(grad_b) # tensor([ 0.2000, -0.1000])Common Mistakes
- Forgetting the bias gradient is simply \(\boldsymbol\delta^{(l)}\) itself, with no dependence on the layer's input — a bias's weight gradient formula genuinely differs from a regular weight's for exactly this reason.
- Getting the outer product's operand order backward — \(\boldsymbol\delta^{(l)}(\mathbf{a}^{(l-1)})^\top\) must produce a matrix with the same shape as \(\mathbf{W}^{(l)}\); swapping the order produces a transposed (and generally wrong-shaped) result.
Interview Relevance
Q: "Why does computing a weight matrix's gradient require both the layer's error signal and its cached input?" Each weight's gradient reflects two things multiplied together: how much error is currently attributed to the output neuron that weight feeds into (captured by \(\boldsymbol\delta\)), and how active the input neuron that weight reads from was during the forward pass (captured by the cached \(\mathbf{a}^{(l-1)}\)). Neither alone is enough — the outer product combines both into the correct per-weight gradient.
Practice Question
If \(\boldsymbol\delta^{(l)} = [0.5]\) (a single output neuron) and \(\mathbf{a}^{(l-1)} = [2.0, -1.0]\), compute the resulting weight gradient matrix's shape and values.
