This note zooms into each of the three layer types individually — what data actually flows through each one, and the specific design decisions you'll make for each when building a real network.
Input Layer
The input layer's "neurons" don't compute anything — they simply hold the raw feature values. Its size is fixed by your data: an image of 28×28 pixels flattened gives an input layer of 784 values; a dataset with 10 numeric features gives an input layer of 10. There are no learnable weights associated with the input layer itself — weights appear on the connections leaving it, toward the first hidden layer.
Hidden Layers
Hidden layers do the actual representation learning — each one transforms its input into a new representation, ideally one that makes the task easier for the next layer. Design choices here include:
| Choice | Consideration |
|---|---|
| Number of hidden layers | More layers → more hierarchical abstraction, but harder to train without mitigations (see Challenges in Deep Learning) |
| Neurons per hidden layer | More neurons → more capacity per layer, more parameters, higher overfitting risk with limited data |
| Activation function | Almost always non-linear (ReLU is the most common modern default) — covered fully next category |
There's no universal formula for "the right" hidden layer sizes — this is one of the hyperparameters tuned empirically (see the Hyperparameter Tuning category), though common practice starts wide-ish and narrows toward the output, or matches known-good architectures for the task type.
Output Layer
The output layer's size and activation are dictated directly by the task — this is one of the few architecture choices that isn't really a free hyperparameter:
| Task | Output Layer Size | Activation |
|---|---|---|
| Binary classification | 1 neuron | Sigmoid |
| Multi-class classification (\(k\) classes) | \(k\) neurons | Softmax |
| Regression (single value) | 1 neuron | None (identity) — or occasionally ReLU if the target is always non-negative |
| Multi-output regression | \(k\) neurons (one per target) | None (identity), typically |
Code — Sizing Each Layer Correctly
import torch.nn as nn
# Example: classifying 28x28 grayscale images into 10 digit classes
model = nn.Sequential(
nn.Flatten(), # input layer: 28*28 = 784 values
nn.Linear(784, 128), nn.ReLU(), # hidden layer 1
nn.Linear(128, 64), nn.ReLU(), # hidden layer 2
nn.Linear(64, 10) # output layer: 10 classes -- no activation here,
# because nn.CrossEntropyLoss applies softmax internally
)Common Mistakes
- Applying softmax explicitly in the model's
forward()and then also usingnn.CrossEntropyLoss(which applies it internally) — this double-applies softmax and silently corrupts training, as flagged already in Cross-Entropy. - Mismatching the output layer size to the task — e.g. using 1 output neuron with softmax (softmax needs at least 2 outputs to be meaningful) or forgetting that binary classification's single sigmoid output represents \(P(\text{class}=1)\), not two separate class probabilities.
Interview Relevance
Q: "How would you size the output layer for a model predicting both a house's price and its number of bedrooms from the same input features?" A multi-output regression head: 2 output neurons (one per target), typically with no activation (identity), since both price and bedroom count are unbounded continuous values (or bounded only by non-negativity, in which case a ReLU output could be considered).
Practice Question
You're building a model to classify an email as spam, promotional, or important (3 mutually exclusive categories). What size and activation should the output layer use?
