This closing note of the Neural Network Fundamentals module traces the weight update rule through a complete, concrete example — one real forward pass, one real backward pass, one real update — tying together everything from the Foundations, Math, and this category into a single worked run.
The Full Cycle, Restated Precisely
Every training step in every network in this hub — from a single Perceptron to a billion-parameter Transformer — executes this exact four-stage cycle. What changes across architectures is only the specific form of \(f\) (the forward computation) and how efficiently \(\nabla_{\mathbf{W}}L\) is computed (backpropagation, applied through whatever operations \(f\) uses).
Complete Worked Example — One Full Update
A single neuron: \(w=0.5, b=0\), input \(x=2\), true label \(y=1\), sigmoid activation, using squared error loss for simplicity, learning rate \(\eta=0.1\).
Forward pass:
Loss:
Backward pass (applying the chain rule from Chain Rule):
Weight update:
The weight increased slightly — makes sense: the true label was 1 but the prediction (0.731) was too low, so increasing the weight (which increases \(z\), which increases \(\hat y\) toward 1) is exactly the right direction.
Code — Verifying This Exact Example with Autograd
import torch
w = torch.tensor(0.5, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)
x = torch.tensor(2.0)
y_true = torch.tensor(1.0)
z = w * x + b
y_pred = torch.sigmoid(z)
loss = (y_true - y_pred) ** 2
loss.backward()
print("dL/dw:", w.grad.item()) # approximately -0.212, matching the manual derivation
lr = 0.1
with torch.no_grad():
w -= lr * w.grad
print("updated w:", w.item()) # approximately 0.5212Why One Update Isn't Enough
A single update nudges the weight only slightly closer to reducing the loss for this one example. Training repeats this cycle across every example (or batch of examples) in the dataset, across many epochs — gradually, the accumulated small steps converge the weights toward values that predict well across the whole training distribution, not just one example. Everything about how to do this efficiently, robustly and at scale is the subject of the Optimization, Training Deep Networks, and Backpropagation categories that follow.
Common Mistakes
- Expecting a single weight update to produce a dramatic accuracy jump — with a small learning rate (by design, for stability), each step makes only a small, incremental improvement; real training requires many such steps.
- Losing track of which quantities are computed during the forward pass (needed later for the backward pass) vs which are only computed during backpropagation itself.
Interview Relevance
Q: "Trace through, with real numbers, what happens during one training step of a neural network." A strong answer walks the same four stages as this note: compute a prediction via forward propagation, compute the loss by comparing to the true label, compute gradients via backpropagation (the chain rule applied layer by layer), then update every weight by subtracting the learning rate times its gradient — and can name the shapes/values involved for a concrete tiny example, not just the abstract formula.
Key Takeaways — Neural Network Fundamentals
- The artificial neuron (weighted sum + activation) is a loose mathematical metaphor for a biological neuron, refined from the McCulloch-Pitts model (fixed, hand-designed) through the Perceptron (learnable, but only linearly separable problems) to the MLP (learnable, non-linear, universal).
- A single Perceptron cannot solve XOR; stacking layers with non-linear activations between them (an MLP) resolves this completely.
- Parameters (weights, biases) are learned by training; hyperparameters (learning rate, architecture size) are chosen by you beforehand.
- The full training cycle — forward propagation, loss, backpropagation, gradient descent weight update — is the same four-stage loop underlying every network in this entire hub, however large or specialized.
Next: Activation Functions covers every non-linearity referenced loosely in this category (ReLU, sigmoid, softmax and more) in full mathematical and practical detail.
Practice Question
Using the same numbers as the worked example above but with true label \(y=0\) instead of \(y=1\), recompute \(\frac{\partial L}{\partial \hat y}\) and predict, without fully recalculating, whether the updated weight should increase or decrease from 0.5.
