Layer Normalization (LayerNorm) normalizes across a single example's own features, entirely independent of the batch — fixing BatchNorm's batch-size dependency, and becoming the standard normalization choice inside Transformer architectures as a direct result.
Formula
\(d\) is the number of features in one example. Critically, \(\mu\) and \(\sigma^2\) here are computed per individual example, across that example's own \(d\) features — with no dependence whatsoever on any other example in the batch, or even on there being a "batch" at all.
The Key Structural Difference from BatchNorm
| BatchNorm | LayerNorm | |
|---|---|---|
| Statistics computed across | All examples in the batch, per feature | All features within one example, independently per example |
| Depends on batch size? | Yes | No |
| Same behavior in training and inference? | No — uses running averages at inference | Yes — identical computation regardless of mode |
| Works with variable-length sequences? | Awkward — batch statistics get complicated with padding | Naturally — each example (e.g. each token's representation) is normalized independently |
Why Transformers Use LayerNorm, Not BatchNorm
Transformers (covered in the Transformers category) process sequences of tokens, often of varying length, and are frequently trained and deployed with widely varying batch sizes — sometimes a batch size of 1 at inference. LayerNorm's complete independence from batch composition and size makes it a natural fit; it behaves identically whether processing a batch of 512 examples during training or a single example during real-time inference, with no separate "running average" bookkeeping required at all.
Numerical Example
One example's 4 features: \([2, 4, 4, 6]\) — notice this is the exact same numbers as BatchNorm's example, but here they represent one example's own features, not one feature across 4 different examples. The computation is identical either way: \(\mu=4\), \(\sigma^2=2\), giving \(\hat x \approx [-1.41, 0, 0, 1.41]\) — the formula is the same; only which values get grouped together for computing \(\mu\) and \(\sigma^2\) has changed.
Code
import torch
import torch.nn as nn
layer_norm = nn.LayerNorm(normalized_shape=64) # normalizes across the last dimension (64 features)
x = torch.randn(32, 64) # batch of 32 examples, 64 features each
output = layer_norm(x)
# Behaves identically regardless of batch size -- unlike BatchNorm
x_single = torch.randn(1, 64)
output_single = layer_norm(x_single) # no train/eval discrepancy, no running averages neededCommon Mistakes
- Assuming LayerNorm always outperforms BatchNorm — for standard CNNs with consistently large batch sizes, BatchNorm often remains competitive or preferable; LayerNorm's advantages are most pronounced specifically for sequence models and variable-batch-size scenarios.
- Forgetting that LayerNorm still needs its learnable \(\gamma\) and \(\beta\) parameters, just like BatchNorm — the normalization step alone isn't the complete picture.
Interview Relevance
Q: "Why do Transformer architectures use LayerNorm instead of BatchNorm?" Transformers process sequences that often vary in length and are trained/deployed across a wide range of batch sizes, including batch size 1 at inference. LayerNorm normalizes each example's own features independently, with no dependence on batch composition or size, and behaves identically in training and inference — avoiding BatchNorm's batch-size sensitivity and its separate running-average bookkeeping, both of which are awkward fits for how Transformers are actually used.
Practice Question
Why doesn't LayerNorm need a separate "running average" mechanism for inference, the way BatchNorm does?
