The autoencoder is the simplest generative-adjacent architecture: a network trained to compress its input down to a smaller representation and then reconstruct the original from that compression — with no labels required at all.
The Structure
The encoder maps the input \(\mathbf{x}\) down to a smaller latent representation \(\mathbf{z}\) (the "bottleneck"); the decoder maps \(\mathbf{z}\) back up to a reconstruction \(\hat{\mathbf{x}}\), trained to match the original input as closely as possible — exactly the reconstruction loss setup from Reconstruction Loss.
Why the Bottleneck Is Essential
If \(\mathbf{z}\) were the same size as \(\mathbf{x}\) (or larger), the network could trivially learn the identity function — just copy the input through unchanged, without learning anything useful. Forcing \(\mathbf{z}\) to be smaller than \(\mathbf{x}\) means the network must discover a compressed, information-dense representation that still captures enough to reconstruct the original — this is exactly what makes the learned \(\mathbf{z}\) useful.
Diagram
The narrow bottleneck forces the network to learn a compressed, information-dense representation of the input.
Code
import torch
import torch.nn as nn
class Autoencoder(nn.Module):
def __init__(self, input_dim, latent_dim):
super().__init__()
self.encoder = nn.Sequential(nn.Linear(input_dim, 128), nn.ReLU(), nn.Linear(128, latent_dim))
self.decoder = nn.Sequential(nn.Linear(latent_dim, 128), nn.ReLU(), nn.Linear(128, input_dim), nn.Sigmoid())
def forward(self, x):
z = self.encoder(x)
x_hat = self.decoder(z)
return x_hat, z
model = Autoencoder(input_dim=784, latent_dim=32) # e.g. MNIST: 784 pixels -> 32-dim compressed code
x = torch.rand(1, 784)
x_hat, z = model(x)
print(z.shape) # (1, 32) -- the compressed representation
print(x_hat.shape) # (1, 784) -- reconstruction, same shape as the originalPractical Uses Beyond Generation
| Use Case | How |
|---|---|
| Dimensionality reduction | Use the learned \(\mathbf{z}\) as a compact feature representation for downstream tasks — a non-linear, learned alternative to PCA |
| Anomaly detection | An autoencoder trained on "normal" data reconstructs normal inputs well but reconstructs unusual/anomalous inputs poorly — a large reconstruction error flags anomalies |
| Pretraining/feature learning | The trained encoder can be reused as a feature extractor for a separate downstream task |
Common Mistakes
- Making the latent dimension too large relative to the input — with too little compression, the network can partially "cheat" toward the trivial identity function rather than learning genuinely useful structure.
- Expecting a plain autoencoder to generate good new samples by feeding it random noise directly — plain autoencoders have no guarantee that the latent space is smooth or well-structured for this purpose; this exact limitation is what motivates the Variational Autoencoder, covered later in this category.
Interview Relevance
Q: "Why does an autoencoder need a bottleneck smaller than the input, and what would happen without one?" Without a smaller bottleneck, the network could trivially learn to copy the input straight through to the output (the identity function), achieving perfect reconstruction without learning anything useful about the data's structure. Forcing the latent representation to be smaller than the input compels the network to discover a genuinely compressed, information-dense encoding.
Practice Question
How could you use a trained autoencoder to detect fraudulent transactions, assuming it was trained only on normal, legitimate transaction data?
