These three terms — epoch, batch, and iteration — describe training at different levels of granularity, and beginners frequently mix them up in ways that make training logs and progress bars confusing to interpret correctly.
Definitions
| Term | Definition |
|---|---|
| Batch (mini-batch) | A subset of the training data used for one weight update (see Mini-Batch Gradient Descent) |
| Batch size | The number of examples in one batch (e.g. 32, 64, 128) |
| Iteration (step) | One weight update — processing exactly one batch through forward pass, backward pass, and optimizer step |
| Epoch | One complete pass through the entire training dataset — many iterations make up one epoch |
The Relationship, as a Formula
The ceiling function accounts for the last, possibly smaller, batch when the dataset size isn't evenly divisible by the batch size.
Numerical Example
A training set has 50,000 examples, batch size is 128:
Training for 20 epochs means \(391 \times 20 = 7{,}820\) total weight updates (iterations) — this is the number that actually matters for a learning rate schedule defined in terms of steps, like One Cycle Learning Rate.
Code — Seeing the Counts Directly
from torch.utils.data import DataLoader
dataset_size = 50000
batch_size = 128
loader = DataLoader(range(dataset_size), batch_size=batch_size)
print(len(loader)) # 391 -- this IS "iterations per epoch"
num_epochs = 20
total_iterations = len(loader) * num_epochs
print(total_iterations) # 7820 -- total weight updates across the whole training runWhere Each Term Shows Up in Practice
- A learning rate schedule like
StepLR(see Step Decay) is typically stepped once per epoch. - A schedule like
OneCycleLRis typically stepped once per iteration (batch), since its phases are defined relative to total training steps. - Training logs commonly report loss both per-iteration (noisy, immediate feedback) and averaged per-epoch (smoother, higher-level trend).
Common Mistakes
- Confusing "iteration" with "epoch" when reading someone else's training configuration — a learning rate schedule or logging interval specified "per step" behaves very differently from one specified "per epoch," and mixing them up silently misconfigures training.
- Assuming batch size doesn't affect how many total iterations a fixed number of epochs represents — a larger batch size means fewer iterations per epoch (and total), which affects how frequently step-based schedules like One Cycle actually adjust the learning rate.
Interview Relevance
Q: "If you double the batch size without changing the number of epochs, what happens to the total number of weight updates?" It roughly halves — since each batch now covers twice as many examples, half as many batches (iterations) are needed to complete one epoch, and therefore half as many total weight updates occur across the same number of epochs. This is part of why learning rate is often scaled up when increasing batch size (see Learning Rate).
Practice Question
A dataset has 12,800 examples and batch size is 256. How many iterations are in one epoch? How many total iterations occur across 15 epochs?
