The chain rule tells you how to differentiate a function that's built by composing other functions — exactly the situation a neural network is in, since it's a chain of layers, each one feeding into the next. The chain rule is the single mathematical fact that makes backpropagation possible.
Formula
To find how \(y\) changes with respect to \(x\), when \(x\) only affects \(y\) indirectly (through \(u\)), you multiply the "local" derivatives along the chain of dependency.
Numerical Example
At \(x=1\): \(u = 3(1)+1 = 4\), so \(\frac{dy}{dx} = 6(4) = 24\). Verify directly: \(y=(3x+1)^2 = 9x^2+6x+1\), so \(\frac{dy}{dx}=18x+6\), and at \(x=1\) that's \(24\) — matches.
A Longer Chain — Like a Neural Network
A network is a chain of many functions: input → layer 1 → activation → layer 2 → activation → ... → loss. For \(n\) composed functions, the chain rule extends naturally:
Each factor is a "local" derivative — how one layer's output changes with respect to its own input — and the full derivative of the loss with respect to any early input is the product of all these local derivatives along the path. This product structure is exactly why the vanishing gradient problem happens: if each local derivative is a fraction less than 1, the product shrinks exponentially with the number of layers.
Code — Chain Rule via Autograd
import torch
x = torch.tensor(1.0, requires_grad=True)
u = 3 * x + 1 # u = g(x)
y = u ** 2 # y = f(u)
y.backward()
print(x.grad) # tensor(24.) -- PyTorch applied the chain rule automaticallyWhere This Shows Up in Deep Learning
Backpropagation is the chain rule, applied systematically from the loss backward through every layer to every weight. Each layer only needs to know how to compute its own "local" derivative (its output with respect to its own input and its own weights) — the chain rule handles stitching all those local derivatives together into the full gradient for every parameter in the network, no matter how deep it is. The full worked example lives in Backpropagation Worked Example.
Common Mistakes
- Forgetting a factor in a long chain — every intermediate function in the composition contributes one multiplicative term; skipping one gives a wrong gradient.
- Assuming the chain rule only applies to single-variable chains — the multivariate version (used throughout deep learning) sums contributions across every path a variable can influence the output through, which is exactly what automatic differentiation tracks via the computational graph.
Interview Relevance
Q: "How does the chain rule relate to backpropagation?" Backpropagation computes the gradient of the loss with respect to every weight by applying the chain rule layer by layer, from the output back to the input. Each layer contributes its own local derivative, and these are multiplied together along the path from that weight to the loss — this is literally what "back-propagating" the gradient means.
Practice Question
Given \(y = \sin(x^2)\), identify the inner and outer functions, then use the chain rule to find \(\frac{dy}{dx}\).
