Multi-Layer Perceptron

The Multi-Layer Perceptron (MLP) stacks several layers of neurons — with non-linear activations between them — to represent functions a single Perceptron fundamentally cannot, including XOR. It is the foundational architecture every more specialized network (CNNs, Transformers) builds on.

Structure

Each layer takes the previous layer's output as its input, applies a matrix multiplication (weights) and bias (see Matrix Multiplication), then a non-linear activation \(\phi\). The final layer typically uses a different activation \(\psi\) suited to the task (e.g. softmax for classification). Layers between the input and output are called hidden layers because their values aren't directly observed — they're intermediate representations the network builds for itself.

Solving XOR with an MLP

A well-known hand-constructed solution uses 2 hidden neurons: hidden neuron 1 computes OR, hidden neuron 2 computes NAND, and the output neuron computes AND of those two hidden outputs. The combination (\(A \text{ OR } B\)) AND (\(A \text{ NAND } B\)) is exactly XOR — the hidden layer built an intermediate representation where the problem becomes linearly separable, even though it wasn't in the raw input space.

Diagram — MLP Architecture

Code

import torch
import torch.nn as nn

class MLP(nn.Module):
    def __init__(self):
        super().__init__()
        self.hidden = nn.Linear(2, 3)   # input layer -> hidden layer (2 inputs, 3 hidden neurons)
        self.output = nn.Linear(3, 1)   # hidden layer -> output layer

    def forward(self, x):
        h = torch.relu(self.hidden(x))     # hidden layer + non-linear activation
        y = torch.sigmoid(self.output(h))   # output layer + sigmoid for binary classification
        return y

model = MLP()
x = torch.tensor([[0.0, 1.0]])
print(model(x))

Why Depth Matters — Connecting Back to Linear Transformations

Recall from Linear Transformations: without a non-linear activation between layers, stacking matrix multiplications collapses into a single linear transformation, no matter how many layers you add. The activation functions between an MLP's layers are exactly what prevent that collapse — they're what let a 2-hidden-layer MLP represent functions (like XOR) that no purely linear model, of any depth, ever could.

Common Mistakes

• Assuming more hidden layers always improve performance — beyond a task's actual complexity, extra depth mainly adds overfitting risk and training difficulty (vanishing gradients) without benefit.
• Forgetting the output layer's activation should match the task: sigmoid for binary classification, softmax for multi-class, no activation (or identity) for unbounded regression outputs.

Interview Relevance

Q: "Why does adding a single hidden layer let an MLP solve XOR, when no single-layer Perceptron can?" The hidden layer learns its own intermediate, non-linear representation of the input. Even though the original two inputs aren't linearly separable for XOR, the hidden layer's output space can be — the network effectively learns a transformation into a space where the final linear output layer's job becomes solvable.

Practice Question

An MLP has an input layer of 10 features, one hidden layer of 20 neurons, and an output layer of 3 classes. How many learnable weights (ignore biases) does this network have in total?

Related DL Notes

• Limitations of Perceptron
• Neural Network Architecture
• Forward Propagation