Once data is cleaned, preprocessing transforms it into the exact numerical format a model can actually consume — normalization, resizing, and tokenization, applied consistently across every data split.
Common Preprocessing Steps by Data Type
| Data Type | Typical Preprocessing | Concept Note |
|---|---|---|
| Images | Resize to a fixed shape, normalize pixel values (often to zero mean, unit variance per channel) | Variance & Standard Deviation |
| Text | Tokenize into subword units, build/apply a vocabulary | Tokenization |
| Tabular/structured features | Scale numerical features, encode categorical features | One-Hot Encoding |
| Audio | Resample to a consistent sample rate, convert to a spectrogram representation | — |
The Critical Rule: Fit Statistics on Training Data Only
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X_train) # compute mean/std ONLY from the training set
X_train_scaled = scaler.transform(X_train)
X_val_scaled = scaler.transform(X_val) # apply the SAME statistics -- never re-fit on validation/test
X_test_scaled = scaler.transform(X_test)This is exactly the data leakage warning already flagged in Dataset Train/Val/Test Split — computing normalization statistics from the combined dataset (including validation/test) leaks information across the split boundary, producing an unrealistically optimistic performance estimate.
Image Normalization — A Common Concrete Example
import torchvision.transforms as transforms
transform = transforms.Compose([
transforms.Resize((224, 224)),
transforms.ToTensor(),
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
# these specific values are ImageNet's channel statistics -- standard when
# using an ImageNet-pretrained model, so inputs match what it expects
])Common Mistakes
- Fitting normalization statistics on the full dataset before splitting, rather than on the training split alone — a subtle but genuine form of data leakage.
- Using different, inconsistent preprocessing between training and later inference/deployment — a mismatch here silently degrades real-world performance, since the model was trained on data preprocessed one way but sees data preprocessed differently in production.
Interview Relevance
Q: "Why must normalization statistics (mean, standard deviation) be computed only from the training set, and applied unchanged to validation and test?" Computing them from the full dataset (including validation/test) leaks information about the held-out data into preprocessing, subtly violating the entire purpose of a held-out set — its statistics shouldn't inform any decision, including preprocessing, made before final evaluation. Fitting statistics on the training set alone, then applying them unchanged to other splits, keeps this boundary honest.
Practice Question
Why is it important that the exact same preprocessing pipeline used during training also be applied at inference time in production?
