"Architecture" describes the overall structure of a neural network — how many layers it has, how wide each one is, and how they connect. This note establishes the vocabulary for talking about network structure precisely, before the next notes zoom into each layer type individually.
The Three Layer Types
| Layer | Role | Count |
|---|---|---|
| Input layer | Receives the raw feature vector — performs no computation itself | Exactly 1 |
| Hidden layer(s) | Learn intermediate representations between input and output | 0 or more (0 = a single-layer Perceptron) |
| Output layer | Produces the network's final prediction, shaped to the task | Exactly 1 |
Depth and Width
| Term | Meaning | Effect of Increasing It |
|---|---|---|
| Depth | Number of layers (usually counting hidden + output) | More capacity for hierarchical, compositional representations; harder to train (vanishing gradients) without careful design |
| Width | Number of neurons in a given layer | More capacity to represent complex functions within one layer; more parameters, more overfitting risk with limited data |
A network with many layers is called "deep" — this is literally where "deep learning" gets its name.
Common Architecture Patterns
- Fully connected (dense): every neuron in one layer connects to every neuron in the next — the MLP pattern from the previous note. Flexible, but doesn't exploit any structure in the input (like an image's spatial layout).
- Convolutional: neurons connect only to a local neighborhood of the previous layer, with shared weights across positions — exploits spatial structure in images (covered fully in the CNN Fundamentals category).
- Recurrent: connections loop back on themselves across time steps — suited to sequential data (covered in the RNN category).
Notation for Describing an Architecture
A common shorthand: "784-128-64-10" describes an MLP with 784 input features, two hidden layers of 128 and 64 neurons, and 10 output classes. This single line fully specifies the shape of every weight matrix in the network — a habit worth building, since it's exactly what you'll need to debug shape errors in real code.
Code — Reading Architecture from a PyTorch Model
import torch.nn as nn
model = nn.Sequential(
nn.Linear(784, 128), nn.ReLU(),
nn.Linear(128, 64), nn.ReLU(),
nn.Linear(64, 10)
)
print(model)
# Sequential(
# (0): Linear(in_features=784, out_features=128, bias=True)
# (1): ReLU()
# (2): Linear(in_features=128, out_features=64, bias=True)
# (3): ReLU()
# (4): Linear(in_features=64, out_features=10, bias=True)
# )
total_params = sum(p.numel() for p in model.parameters())
print(total_params) # counts every weight and bias across all layersCommon Mistakes
- Assuming architecture design is arbitrary or purely a matter of trial and error — matching architecture to data structure (convolutional for images, recurrent/attention-based for sequences) is a core engineering decision with strong theoretical and empirical justification, covered in depth in later categories.
- Conflating "deep" with "good" — depth without appropriate techniques (residual connections, normalization, careful initialization) can make training harder, not better, due to vanishing/exploding gradients.
Interview Relevance
Q: "What's the difference between making a network 'deeper' vs 'wider,' and what are the tradeoffs?" Deeper means more layers, enabling more hierarchical/compositional representations but increasing training difficulty (vanishing gradients) without mitigations like residual connections. Wider means more neurons per layer, increasing per-layer capacity and parameter count, with a more direct overfitting risk on limited data but generally easier optimization than comparable added depth.
Practice Question
Describe the architecture "3072-512-256-100" in words: what could the input and output layer sizes plausibly represent for an image classification task?
