Before any training loop runs, a dataset must be split into three distinct parts — train, validation, and test — each serving a purpose the others cannot substitute for. Mixing up their roles is one of the most common ways a deep learning project silently produces misleading results.
The Three Splits
| Split | Used For | How Often Touched |
|---|---|---|
| Training set | Computing gradients and updating weights | Every batch, every epoch |
| Validation set | Monitoring generalization during training — tuning hyperparameters, deciding when to stop | Periodically during training (e.g. once per epoch) |
| Test set | A final, one-time estimate of real-world performance | Once, after all training and tuning decisions are finalized |
Why Three Sets, Not Two
It's tempting to think "train on most of the data, test on the rest" is enough — but if you use the same held-out set to both tune hyperparameters (learning rate, architecture, when to stop) and report final performance, you've implicitly let information from that set leak into your modeling decisions. Your reported performance becomes an optimistic estimate, not a fair one. The validation set absorbs all that tuning-related "peeking"; the test set stays completely untouched until the very end, giving an honest final estimate.
Typical Split Ratios
| Dataset Size | Common Split |
|---|---|
| Small (thousands of examples) | 60% train / 20% validation / 20% test, or similar |
| Large (millions of examples) | 98% train / 1% validation / 1% test — even 1% is often plenty of examples at this scale |
As dataset size grows, the validation and test sets don't need to grow proportionally — a few thousand examples is usually enough to get a statistically reliable performance estimate, regardless of how many millions of examples the training set has.
Code
from sklearn.model_selection import train_test_split
# First split off the test set
X_temp, X_test, y_temp, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Then split the remainder into train and validation
X_train, X_val, y_train, y_val = train_test_split(X_temp, y_temp, test_size=0.25, random_state=42)
# 0.25 of the remaining 80% = 20% of the original data -- giving a 60/20/20 split overall
print(len(X_train), len(X_val), len(X_test))import torch
from torch.utils.data import random_split
dataset = MyDataset() # a full PyTorch Dataset
train_size = int(0.6 * len(dataset))
val_size = int(0.2 * len(dataset))
test_size = len(dataset) - train_size - val_size
train_ds, val_ds, test_ds = random_split(dataset, [train_size, val_size, test_size])A Critical Detail: Splitting Before Preprocessing
Any preprocessing statistics (feature means/standard deviations for standardization, vocabulary for tokenization) must be computed only from the training set, then applied unchanged to validation and test — computing them from the full dataset before splitting leaks information from validation/test into training, a subtle but real form of the same "peeking" problem this whole splitting discipline exists to prevent.
Common Mistakes
- Tuning hyperparameters against the test set instead of the validation set — this quietly turns the test set into a second validation set, and your final reported number stops being an honest estimate of real-world performance.
- Computing normalization statistics (mean, std) from the full dataset before splitting — this is a data leakage bug, not just poor practice; it makes validation/test performance artificially optimistic.
- Splitting time-series or sequential data randomly, ignoring temporal order — for many real-world problems, validation/test data should come chronologically after training data, or the split doesn't reflect how the model will actually be used.
Interview Relevance
Q: "Why do you need a separate validation set if you already have a test set?" The validation set is where all hyperparameter tuning and "when to stop training" decisions happen — using it repeatedly during development doesn't compromise the fairness of a final evaluation. The test set must stay completely untouched until the very end; if it were used for any tuning decisions, its final performance number would be an overly optimistic estimate of how the model performs on truly unseen data.
Practice Question
A dataset has 100,000 examples. You use 70,000 for training, and want a validation set large enough for reliable early-stopping decisions, plus a test set for final reporting. Propose a reasonable split and justify it.
