Good experimental design — planning experiments carefully before running them — is what ultimately makes research results trustworthy, tying together the baselines, ablations, and reproducibility practices covered throughout this category into one coherent, rigorous methodology.
The Core Principles of Sound Experimental Design
| Principle | What It Means in Practice |
|---|---|
| Control variables | Change only the specific factor being tested; keep everything else identical between compared conditions |
| Define hypotheses before running experiments | State what result is expected and why, before seeing the actual outcome — avoids the temptation to retroactively rationalize whatever result happens to occur |
| Plan the evaluation protocol upfront | Decide which metrics and comparisons matter before seeing results, directly echoing DL Problem Definition's emphasis on this same principle |
| Account for randomness | Run enough seeds/trials to distinguish genuine effects from noise, rather than drawing conclusions from a single run |
A Concrete Example: Designing a Fair Comparison
# A well-designed experiment for comparing two optimizers
experimental_design = {
"hypothesis": "Optimizer B converges faster than Optimizer A on this specific task",
"controlled_variables": ["model architecture", "dataset", "batch size", "number of epochs",
"random seed set (same seeds used for both conditions)"],
"varied_variable": "optimizer choice (A vs B)",
"tuning_protocol": "each optimizer gets an EQUAL hyperparameter search budget",
"num_seeds_per_condition": 5,
"evaluation_metric": "validation loss at a fixed compute budget",
"significance_test": "paired t-test across matched seeds"
}Every element here exists specifically to ensure that if a difference is observed, it can be confidently attributed to the one varied factor (optimizer choice) rather than some other confounding difference between the two conditions.
Avoiding "p-hacking" and Post-Hoc Rationalization
A subtle but real risk: running many experiments, and after the fact, selectively reporting or emphasizing whichever comparisons happened to show a favorable result — sometimes called p-hacking in the broader statistics literature. Defining the specific hypothesis and evaluation plan before running experiments (as in the code example above) is a direct, structural defense against this kind of unintentional (or intentional) result-shopping.
Why This Ties the Whole Category Together
Baselines (Baselines in Research), ablations (Ablation Studies), reproducibility practices (Reproducibility), and statistical rigor (Statistical Significance) are all, fundamentally, specific applications of sound experimental design principles — controlling variables, planning evaluation upfront, and accounting for randomness — applied to the specific context of deep learning research.
Common Mistakes
- Deciding which comparisons or metrics to report only after seeing results, rather than planning the evaluation protocol upfront — this risks selectively favoring whichever framing happens to look most impressive, rather than reflecting a genuinely planned, honest evaluation.
- Varying multiple experimental factors simultaneously when trying to isolate the effect of just one — this confounds the results, making it impossible to attribute an observed difference to any single specific cause.
Interview Relevance
Q: "Why should the evaluation metrics and comparisons for an experiment be decided before running it, rather than chosen afterward based on the results?" Deciding evaluation criteria retroactively, after seeing results, creates a real risk of selectively emphasizing whichever metric or comparison happens to show the most favorable outcome for a preferred hypothesis or method — sometimes called p-hacking or result-shopping. Committing to the hypothesis, metrics, and comparison protocol before running the experiment is a structural safeguard against this bias, ensuring the resulting conclusions genuinely reflect the evidence rather than a post-hoc narrative constructed to fit whatever outcome occurred.
Practice Question
Why does comparing two methods using the exact same set of random seeds for each (a "paired" design) allow for a more statistically powerful comparison than using different, unrelated seeds for each?
