Backpropagation (Intro)

This note introduces backpropagation at a conceptual level — enough to complete the training-loop picture in this category. The full mathematical derivation, with a complete numerical example, gets its own dedicated category right after this one.

The One-Sentence Definition

Backpropagation is the algorithm that computes the gradient of the loss with respect to every weight in the network, efficiently, by applying the chain rule backward from the loss through each layer in reverse order.

Why "Backward"?

Forward propagation computes left to right: input → layer 1 → layer 2 → ... → output → loss. Backpropagation computes right to left: starting from the loss, it works out how sensitive the loss is to the last layer's outputs, then uses the chain rule (see Chain Rule) to work out how sensitive the loss is to the last layer's weights and to the previous layer's outputs, then repeats this one layer earlier, and so on, until it reaches the very first layer.

Why Not Compute Each Gradient Independently?

You could, in principle, compute the gradient for each weight from scratch using the chain rule directly — but this repeats a huge amount of work, since many weights share overlapping paths to the loss. Backpropagation is efficient specifically because it reuses intermediate gradient computations (computed once per layer, working backward) across every weight that depends on them — turning what would be an intractable amount of redundant computation into something that costs roughly the same as one additional forward pass.

Code — The Backward Pass in One Line

import torch
import torch.nn as nn

model = nn.Sequential(nn.Linear(2, 3), nn.ReLU(), nn.Linear(3, 1))
x = torch.tensor([[1.0, 2.0]])
y_true = torch.tensor([[5.0]])

y_pred = model(x)                          # forward pass
loss = nn.MSELoss()(y_pred, y_true)         # compute the loss

loss.backward()   # backward pass -- computes gradients for EVERY parameter in the model at once

for name, param in model.named_parameters():
    print(name, param.grad.shape)   # every weight and bias now has a populated .grad

What Happens Next — The Missing Piece

Backpropagation computes gradients — it doesn't, by itself, change any weights. The actual weight update is a separate step, covered in the next note and the full Optimization category: \(\mathbf{w} \leftarrow \mathbf{w} - \eta\nabla L(\mathbf{w})\).

Common Mistakes

• Believing .backward() updates the model's weights — it only computes and stores gradients in each parameter's .grad attribute; a separate optimizer.step() call actually applies the update.
• Treating backpropagation as conceptually distinct from the chain rule — it's not a different mathematical idea, just a specific, efficient bookkeeping strategy for applying the chain rule across a whole network at once.

Interview Relevance

Q: "In one sentence, what problem does backpropagation solve that naive gradient computation wouldn't?" It computes the gradient of the loss with respect to every weight in the network efficiently, by reusing shared intermediate computations while working backward through the layers — avoiding the massive redundant recomputation that computing each weight's gradient independently from scratch would require.

Practice Question

In your own words, explain why backpropagation needs the values computed during the forward pass (like each layer's activations) in order to compute gradients during the backward pass.

Related DL Notes

• Chain Rule
• Gradient Descent (Intro)
• Why Backpropagation