Calculus is the mathematics of change — and a neural network learns by repeatedly asking "if I change this weight slightly, how does the loss change?" That question only makes sense once you have a precise idea of a function and how it behaves as its input changes by a small amount — a limit.
Functions — The Object Calculus Studies
A function \(f(x)\) maps an input to an output. In deep learning, the most important function is the loss function \(L(\mathbf{w})\) — it maps the network's entire set of weights \(\mathbf{w}\) to a single number describing how wrong the network currently is. Training is the process of finding weights that make \(L(\mathbf{w})\) as small as possible, and calculus is the toolkit for finding that minimum efficiently.
Limits — Formalizing "As x Gets Arbitrarily Close"
This reads: "as \(x\) gets arbitrarily close to \(a\), \(f(x)\) gets arbitrarily close to \(L\)." Limits let mathematicians talk rigorously about instantaneous rates of change — a derivative (the next note) is formally defined as a limit.
Numerical Example
As \(x\) approaches 3 from either side (2.9, 2.99, 2.999, ... or 3.1, 3.01, 3.001, ...), \(f(x)\) approaches 9. For smooth functions like this one, the limit simply equals \(f(3)\) — but limits become essential for functions that are undefined or discontinuous at a specific point, which is exactly how a derivative is defined at every point.
Code — Approximating a Limit Numerically
def f(x):
return x ** 2
a = 3
for h in [0.1, 0.01, 0.001, 0.0001]:
print(f"x={a+h:.4f}, f(x)={f(a+h):.6f}")
# As h shrinks, f(a+h) converges toward f(3) = 9Why Deep Learning Needs Calculus at All
A neural network's loss function typically depends on millions of weights simultaneously, and there's no formula you can just "solve" to find the best weights directly (unlike simple linear regression's normal equation — see Matrix Inverse). Instead, training works by repeatedly asking: which direction should each weight move, right now, to reduce the loss? That direction is exactly what a derivative tells you — this whole Calculus for DL category builds toward the gradient, and ultimately toward backpropagation, which computes that direction efficiently for every weight in the network at once.
Common Mistakes
- Treating calculus as separate "theory" from deep learning practice — every call to
loss.backward()in PyTorch is executing calculus (the chain rule) automatically; understanding the math tells you what that call is actually doing. - Assuming a limit always equals the function's value at that point — this holds for smooth, continuous functions (most functions you'll use), but is exactly the subtlety that makes derivatives at "kink" points (like ReLU at 0) worth discussing explicitly later.
Interview Relevance
Q: "Why does training a neural network require calculus at all?" Because there's no closed-form solution for the weights that minimize a deep network's loss — training instead relies on iteratively computing the direction (via derivatives/gradients) that reduces the loss, then taking a small step in that direction, repeated many times.
Practice Question
Evaluate \(\lim_{x \to 2} (3x + 1)\) by reasoning about what value \(3x+1\) approaches as \(x\) gets close to 2.
