A computational graph is a diagram of every individual operation a function performs, connected in the order they execute. It's not just a teaching tool — it's literally how PyTorch and TensorFlow track operations internally to compute gradients automatically.
Building a Computational Graph
Take a small function: \(f(x, y) = (x + y) \times y\). Break it into individual operations:
The forward pass flows left to right through this graph; backpropagation flows right to left, applying the chain rule at every node.
Forward Pass vs Backward Pass
| Pass | Direction | What Happens |
|---|---|---|
| Forward pass | Inputs → output | Compute each node's value in order, ending at the final output (e.g. the loss) |
| Backward pass | Output → inputs | Starting from the output, apply the chain rule at each node to compute how the final output depends on every earlier value, all the way back to the original inputs |
Numerical Walkthrough
With \(x=2, y=3\): forward pass gives \(a = 2+3=5\), \(f = 5\times3=15\). Backward pass: \(\frac{\partial f}{\partial a}=y=3\), \(\frac{\partial f}{\partial y}\) (direct path)\(=a=5\), \(\frac{\partial a}{\partial x}=1\), \(\frac{\partial a}{\partial y}\) (through \(a\))\(=1\). By the chain rule, \(\frac{\partial f}{\partial x} = \frac{\partial f}{\partial a}\cdot\frac{\partial a}{\partial x} = 3\times1=3\). Since \(y\) affects \(f\) through two paths (directly, and through \(a\)), its total gradient sums both: \(\frac{\partial f}{\partial y} = 5 + (3\times1) = 8\).
Code — PyTorch Builds This Graph Automatically
import torch
x = torch.tensor(2.0, requires_grad=True)
y = torch.tensor(3.0, requires_grad=True)
a = x + y # PyTorch records this operation in the graph
f = a * y # and this one
f.backward() # walks the recorded graph backward, applying the chain rule at each node
print(x.grad, y.grad) # tensor(3.) tensor(8.) -- matches the manual calculation aboveWhy This Matters for Deep Learning
A neural network's forward pass — input through every layer to the loss — is a (much larger) computational graph. PyTorch builds this graph dynamically as your code executes (this is what "define-by-run" / eager mode means), then .backward() walks it in reverse, applying the chain rule at every single node automatically. This is the concrete mechanism behind automatic differentiation, and it's what makes backpropagation practical for networks with millions of operations — no one writes out the chain rule by hand.
Common Mistakes
- Thinking of the computational graph as a static diagram you draw once — PyTorch's graph is rebuilt fresh on every forward pass by default, which is what makes dynamic control flow (loops, conditionals depending on data) possible in the first place.
- Forgetting that a variable used in multiple places (like \(y\) in the example) accumulates gradient contributions from every path — missing this is a common source of manually-computed-gradient errors.
Interview Relevance
Q: "What is a computational graph, and why does PyTorch build one?" It's a graph of every elementary operation performed during the forward pass, recording how each value depends on earlier ones. PyTorch builds it automatically so that .backward() can traverse it in reverse, applying the chain rule at each node to compute gradients for every parameter — this is the mechanism of automatic differentiation.
Practice Question
Draw the computational graph for \(f(x) = (x^2 + 1) \times x\), labeling each intermediate node, then trace the backward pass to find \(\frac{df}{dx}\) at \(x=2\).
