The forward pass was already introduced in Forward Propagation as how a network produces a prediction. Here, the focus shifts to a detail that matters specifically for backpropagation: what needs to be remembered during the forward pass so the backward pass can use it later.
The Forward Pass, Revisited
What Must Be Cached, and Why
| Cached Value | Why the Backward Pass Needs It |
|---|---|
| Each layer's input, \(\mathbf{a}^{(l-1)}\) | Needed to compute \(\frac{\partial L}{\partial \mathbf{W}^{(l)}}\), since the weight gradient formula involves the layer's input directly (see Gradient Calculation) |
| Each layer's pre-activation, \(\mathbf{z}^{(l)}\) | Needed to compute the activation function's own derivative, \(\phi'(\mathbf{z}^{(l)})\) — e.g. sigmoid's derivative depends on its own output, ReLU's derivative depends on whether \(z>0\) |
Without these cached values, the backward pass would need to recompute them by re-running parts of the forward pass — wasting exactly the computational efficiency backpropagation is designed to provide. This is why training uses meaningfully more memory than pure inference: every intermediate activation across every layer must be kept in memory until the backward pass has used it.
Code — Manually Tracking What Gets Cached
import torch
class ManualLinearSigmoid:
def forward(self, x, W, b):
z = x @ W.T + b
a = torch.sigmoid(z)
self.cache = (x, z, a) # exactly what the backward pass will need
return a
layer = ManualLinearSigmoid()
x = torch.tensor([[1.0, 2.0]])
W = torch.tensor([[0.5, -0.3]])
b = torch.tensor([0.1])
output = layer.forward(x, W, b)
print(layer.cache) # (input, pre-activation, activation) -- all needed laterIn practice, PyTorch's autograd handles this caching automatically as part of building the computational graph (see Computational Graphs) — every operation you perform on a tensor with requires_grad=True silently retains whatever values its backward computation will need.
Common Mistakes
- Assuming inference (prediction-only, no training) needs the same memory as training — since no backward pass will run, inference code can discard intermediate activations immediately, and frameworks provide specific modes (like PyTorch's
torch.no_grad()) to skip this caching entirely, saving substantial memory. - Manually re-implementing a layer's forward pass without caching what its backward pass will need — a common bug when writing custom autograd functions from scratch.
Interview Relevance
Q: "Why does training a neural network use more memory than just running inference on it?" Training's backward pass needs each layer's cached inputs and pre-activations from the forward pass to compute gradients. Inference never runs a backward pass, so it can discard each layer's intermediate values immediately after computing the next layer — this is exactly why inference-only code (e.g. wrapped in torch.no_grad()) uses substantially less memory than training.
Practice Question
Why does the backward pass need a layer's pre-activation value \(\mathbf{z}^{(l)}\) specifically, rather than just its post-activation output \(\mathbf{a}^{(l)}\)?
