A real project runs many training experiments — different hyperparameters, architectures, and data versions — and experiment tracking is the discipline of systematically recording each run so results remain comparable and reproducible.
What to Track for Every Experiment
| Category | Examples |
|---|---|
| Configuration | Hyperparameters, architecture choice, random seed |
| Data | Dataset version/hash used for this specific run |
| Code | Git commit hash, so the exact code that produced this result can be recovered later |
| Metrics | Training/validation loss and metrics, logged throughout training, not just the final value |
| Artifacts | The resulting model checkpoint, plots, and any other outputs |
Why Ad Hoc Tracking (Spreadsheets, Memory) Breaks Down
With just a handful of experiments, informally remembering "run 3 with a lower learning rate" might work — but as experiment count grows into the dozens or hundreds (routine during hyperparameter search, see the Hyperparameter Tuning category), manually tracking configuration and results becomes unreliable and error-prone, and questions like "which exact run produced our best model, and with what config?" become genuinely hard to answer without a system.
Code — Tracking Experiments Programmatically
import mlflow # covered in full in the next note
with mlflow.start_run():
mlflow.log_params({"learning_rate": 0.001, "batch_size": 32, "optimizer": "adam"})
for epoch in range(num_epochs):
train_loss = train_one_epoch(model, train_loader, optimizer)
val_loss, val_acc = evaluate(model, val_loader)
mlflow.log_metrics({"train_loss": train_loss, "val_loss": val_loss, "val_acc": val_acc}, step=epoch)
mlflow.pytorch.log_model(model, "model")Every run is now automatically recorded with its exact configuration, per-epoch metrics, and resulting model artifact — queryable and comparable later without relying on memory or manual notes.
Comparing Runs Systematically
A dedicated experiment tracking tool provides a UI or query interface to sort and filter runs by any logged metric or parameter — directly answering questions like "show me every run with learning rate below 0.001, sorted by validation accuracy" instantly, something a manual spreadsheet approach struggles to support well at scale.
Common Mistakes
- Relying on informal notes, filenames, or memory to track experiment configurations — this breaks down quickly as the number of experiments grows, and makes past results hard to reliably reproduce or compare.
- Logging only the final metric value rather than metrics throughout training — this loses valuable information for diagnosing training dynamics (e.g. when overfitting began) after the fact.
Interview Relevance
Q: "Why does experiment tracking become essential once a team is running systematic hyperparameter search, rather than just a few manual training runs?" Systematic search (grid search, random search, Bayesian optimization) can easily produce dozens to hundreds of runs — manually tracking each run's exact configuration and results becomes unreliable and error-prone at this scale, and answering "which configuration actually produced the best result" becomes genuinely hard without a system. Dedicated tracking tools log configuration, metrics, and artifacts automatically for every run, making them reliably queryable and comparable later.
Practice Question
Why is it valuable to log a training run's exact git commit hash alongside its hyperparameters and results?
