Batch Normalization

Batch Normalization (BatchNorm) normalizes each feature (channel) using statistics computed across the entire mini-batch — the original, most widely known normalization technique, and the direct reason "train mode" and "eval mode" behave differently for so many layers.

Formula

\(m\) is the batch size, and \(\mu_B, \sigma_B^2\) are computed per feature/channel, across every example currently in the mini-batch. Every example in the batch is normalized using the exact same \(\mu_B\) and \(\sigma_B^2\) — which is precisely why BatchNorm's behavior depends on batch size and composition.

The Critical Training-vs-Inference Split

This is exactly why forgetting model.eval() before validation (flagged repeatedly in Validation Loop) is such a common and consequential bug for any model using BatchNorm — without it, evaluation would use the current (possibly tiny, unrepresentative) batch's own statistics instead of the stable running averages learned throughout training.

Numerical Example

A single feature's values across a batch of 4 examples: \([2, 4, 4, 6]\). \(\mu_B = \frac{2+4+4+6}{4}=4\). \(\sigma_B^2 = \frac{(-2)^2+0^2+0^2+2^2}{4}=\frac{8}{4}=2\). Normalized: \(\hat x = \frac{[2,4,4,6]-4}{\sqrt{2+\epsilon}} \approx [-1.41, 0, 0, 1.41]\) — this batch's own mean and spread have been standardized away.

Code

import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(20, 64),
    nn.BatchNorm1d(64),   # normalizes each of the 64 features across the batch dimension
    nn.ReLU(),
    nn.Linear(64, 10)
)

model.train()
x = torch.randn(32, 20)   # batch of 32
print(model(x).shape)   # BatchNorm uses THIS batch's own statistics here

model.eval()
x_single = torch.randn(1, 20)   # a batch of just 1 example
print(model(x_single).shape)   # BatchNorm uses its LEARNED RUNNING AVERAGES here, not this one example's stats

The Small-Batch Weakness

Because BatchNorm's training-time statistics come directly from the current mini-batch, a very small batch size produces noisy, unreliable estimates of \(\mu_B\) and \(\sigma_B^2\) — undermining the whole technique's stability benefit. This weakness is exactly what motivates Group Normalization and Layer Normalization (covered in upcoming notes), both of which compute statistics independently of batch size entirely.

Common Mistakes

• Forgetting model.eval() before validation/inference with a BatchNorm-containing model — as emphasized above, this is one of the most consequential and common bugs specific to this technique.
• Using BatchNorm with very small batch sizes (e.g. batch size 1 or 2) without recognizing the resulting statistics will be noisy and unreliable.
• Using BatchNorm in architectures where batch composition is unusual or inconsistent (e.g. certain reinforcement learning setups) without considering whether Layer or Group Normalization would be more robust.

Interview Relevance

Q: "Why does BatchNorm behave differently during training and inference?" During training, it normalizes using the current mini-batch's own mean and variance, which change every step and depend on the specific examples currently in the batch. During inference, using a live batch's statistics would be unreliable (especially for a batch of size 1) and non-deterministic — so BatchNorm instead uses a running average of the statistics accumulated throughout training, giving stable, deterministic behavior regardless of what's being evaluated.

Practice Question

Why might BatchNorm perform poorly if trained with a batch size of just 2 or 4 examples?

Related DL Notes

• Why Normalization
• Layer Normalization
• Group Normalization