Implementation exercises for core neural network mechanics — building a perceptron and a small MLP from scratch in plain NumPy, to genuinely understand forward propagation and backpropagation before relying on a framework to handle it.
🟢 Problem 1: Implement a single perceptron from scratch
Task: Implement a perceptron that takes a NumPy input vector, computes the weighted sum plus bias, and applies a step activation function. Train it on the AND logic gate (4 examples).
import numpy as np
def perceptron_train(X, y, lr=0.1, epochs=20):
weights = np.zeros(X.shape[1])
bias = 0.0
for epoch in range(epochs):
for xi, target in zip(X, y):
pred = 1 if np.dot(xi, weights) + bias > 0 else 0
error = target - pred
weights += lr * error * xi
bias += lr * error
return weights, bias
X = np.array([[0,0],[0,1],[1,0],[1,1]])
y = np.array([0,0,0,1]) # AND gate
w, b = perceptron_train(X, y)
print(f"weights={w}, bias={b}")
for xi in X:
pred = 1 if np.dot(xi, w) + b > 0 else 0
print(f"{xi} -> {pred}")Hint if stuck: The perceptron learning rule updates weights only when a prediction is wrong — if error == 0, weights don't change for that example.
🟡 Problem 2: Implement forward propagation for a 2-layer MLP, by hand
Task: Given a 2-input, 3-hidden-unit, 1-output network with sigmoid activations, implement the forward pass manually using matrix operations. Verify your output shape at each step.
def sigmoid(z): return 1 / (1 + np.exp(-z))
def forward_pass(X, W1, b1, W2, b2):
Z1 = X @ W1 + b1 # (batch, 2) @ (2, 3) -> (batch, 3)
A1 = sigmoid(Z1)
Z2 = A1 @ W2 + b2 # (batch, 3) @ (3, 1) -> (batch, 1)
A2 = sigmoid(Z2)
return A1, A2
np.random.seed(0)
W1 = np.random.randn(2, 3) * 0.1
b1 = np.zeros(3)
W2 = np.random.randn(3, 1) * 0.1
b2 = np.zeros(1)
X = np.array([[0.5, 0.8]])
A1, A2 = forward_pass(X, W1, b1, W2, b2)
print(f"Hidden activations: {A1}")
print(f"Output: {A2}")Hint if stuck: Track each matrix's shape at every line — a shape mismatch almost always reveals exactly where the bug is.
🔴 Problem 3: Implement full backpropagation for the same 2-layer MLP
Task: Extend Problem 2 to compute gradients for W1, b1, W2, b2 given a target output, using mean squared error loss. Then implement one gradient descent update step.
def backward_pass(X, y, A1, A2, W2):
m = X.shape[0]
dZ2 = (A2 - y) * A2 * (1 - A2) # dL/dZ2, sigmoid derivative included
dW2 = A1.T @ dZ2 / m
db2 = np.sum(dZ2, axis=0) / m
dA1 = dZ2 @ W2.T
dZ1 = dA1 * A1 * (1 - A1) # sigmoid derivative for hidden layer
dW1 = X.T @ dZ1 / m
db1 = np.sum(dZ1, axis=0) / m
return dW1, db1, dW2, db2
y = np.array([[1.0]])
dW1, db1, dW2, db2 = backward_pass(X, y, A1, A2, W2)
lr = 0.1
W1 -= lr * dW1; b1 -= lr * db1
W2 -= lr * dW2; b2 -= lr * db2
print("Weights updated via one gradient descent step")Hint if stuck: Work backward one layer at a time — compute \(dZ2\) first (it only needs the output layer), then use it to compute \(dW2\)/\(db2\), then propagate to \(dA1\), then \(dZ1\), then \(dW1\)/\(db1\). Trying to derive all four gradients simultaneously is where most people get lost.
🟡 Problem 4: Verify your gradients numerically
Task: Implement numerical gradient checking — perturb one weight slightly, measure the resulting change in loss, and confirm it approximately matches your analytically computed gradient.
def numerical_gradient_check(X, y, W1, b1, W2, b2, epsilon=1e-4):
def compute_loss(W1, b1, W2, b2):
_, A2 = forward_pass(X, W1, b1, W2, b2)
return np.mean((A2 - y) ** 2)
i, j = 0, 0 # check one specific weight
W1_plus = W1.copy(); W1_plus[i, j] += epsilon
W1_minus = W1.copy(); W1_minus[i, j] -= epsilon
numerical_grad = (compute_loss(W1_plus, b1, W2, b2) - compute_loss(W1_minus, b1, W2, b2)) / (2 * epsilon)
print(f"Numerical gradient: {numerical_grad:.6f}")
print(f"Analytical gradient (from backward_pass): {dW1[i, j]:.6f}")
# These two values should be very close if your backward pass is implemented correctlyHint if stuck: This is exactly how you'd debug a suspected bug in a real backprop implementation — if the numerical and analytical gradients diverge significantly, the bug is in your backward pass math, not somewhere else.
