Practical guidance for choosing batch size — building on the tradeoffs already outlined in Mini-Batch Gradient Descent.
Common Starting Values
Batch sizes are almost always chosen as powers of 2 (32, 64, 128, 256, ...) — this isn't a mathematical requirement, but it aligns well with how GPU memory and parallel compute are organized, often producing modestly better hardware utilization than an arbitrary batch size.
The Linear Scaling Rule
When increasing batch size, a common practical heuristic — directly connected to Learning Rate's note on batch size/learning rate interaction — is to scale the learning rate up proportionally, since larger batches produce less noisy (more "confident") gradient estimates, tolerating and often benefiting from a correspondingly larger step size.
Memory as the Practical Ceiling
# A common practical debugging pattern: find the largest batch size that fits
batch_size = 256
try:
train_one_step(model, batch_size)
except torch.cuda.OutOfMemoryError:
print(f"batch_size={batch_size} too large for available GPU memory")
# halve it and retry, or use gradient accumulation (see PyTorch Custom Training Loops)In practice, batch size is frequently chosen based on what fits in available GPU memory as much as by any theoretical optimum — gradient accumulation (from PyTorch Custom Training Loops) is the standard workaround when a desired effective batch size exceeds available memory.
The Generalization Tradeoff
Some research has found very large batch sizes can, in certain settings, lead to slightly worse generalization than smaller ones, potentially related to smaller batches' inherent noise acting as a mild implicit regularizer (see Stochastic Gradient Descent's discussion of noise helping escape shallow local minima) — this effect isn't universal, but it's a real consideration worth validating empirically for a specific task rather than assuming "bigger batch is always better."
Common Mistakes
- Increasing batch size without correspondingly adjusting the learning rate — this can leave training either overly slow (learning rate now too small relative to the less-noisy gradient) or fail to realize the full benefit of the larger batch.
- Assuming the largest batch size that fits in memory is automatically the best choice — memory capacity and generalization/convergence quality are two separate considerations, not the same thing.
Interview Relevance
Q: "Why does the linear scaling rule suggest increasing the learning rate when you increase batch size?" A larger batch produces a less noisy, more "confident" estimate of the true gradient (averaged over more examples), tolerating — and often benefiting from — a proportionally larger step size without becoming unstable. Keeping the learning rate fixed while increasing batch size can leave training effectively slower than necessary, since the more reliable gradient estimate isn't being exploited with a correspondingly larger step.
Practice Question
If a model was tuned well with batch size 32 and learning rate 0.001, what learning rate would the linear scaling rule suggest for batch size 128?
