If the expected value tells you the center of a distribution, variance and standard deviation tell you its spread — how far values typically land from that center. This spread is exactly what feature normalization and batch normalization are designed to control.
Formulas
Variance is the expected squared deviation from the mean — squaring ensures negative and positive deviations don't cancel out. Standard deviation \(\sigma\) is the square root of variance, which brings the units back to match the original data (variance is in squared units, which is harder to interpret directly).
Numerical Example
Data: \([2, 4, 4, 4, 5, 5, 7, 9]\). Mean \(=5\). Deviations from the mean: \([-3,-1,-1,-1,0,0,2,4]\). Squared deviations: \([9,1,1,1,0,0,4,16]\). Variance \(= \frac{9+1+1+1+0+0+4+16}{8} = \frac{32}{8}=4\). Standard deviation \(=\sqrt{4}=2\).
Code
import numpy as np
data = np.array([2, 4, 4, 4, 5, 5, 7, 9])
print(np.var(data)) # 4.0
print(np.std(data)) # 2.0Why This Matters for Feature Scaling
Features with very different scales (e.g. "age in years" vs "income in dollars") have very different variances, which can make gradient descent converge unevenly — the loss surface becomes elongated in some directions and steep in others. Standardizing a feature to zero mean and unit variance, \(z=\frac{x-\mu}{\sigma}\), is exactly what removes this imbalance before training, and it's the same formula batch normalization applies internally to layer activations (see Batch Normalization).
Population vs Sample Variance
When estimating variance from a data sample (which describes essentially all real training data), dividing by \(n-1\) instead of \(n\) corrects a small bias — this is why np.var(data, ddof=1) exists as an option, though deep learning code frequently just uses the population formula (\(n\)) for simplicity in large-batch settings, where the difference is negligible.
Common Mistakes
- Comparing variances across features measured in different units directly — standard deviation is more interpretable since it's in the same units as the original feature, and normalized comparisons (like the coefficient of variation) are needed for cross-feature comparison.
- Forgetting to compute normalization statistics (mean, std) only from the training set, then applying those same fixed values to the validation/test set — computing them separately per split leaks information and gives an unrealistic performance estimate.
Interview Relevance
Q: "Why do we standardize features before training a neural network?" Features on very different scales create a loss surface with very different curvature in different directions, making gradient descent converge slowly or unevenly. Standardizing every feature to zero mean and unit variance keeps gradients well-scaled across all input dimensions, letting a single learning rate work well for the whole network.
Practice Question
Two features have the same mean but standard deviations of 1 and 50 respectively. Explain, using the standardization formula, what problem this could cause if left unaddressed before training.
