This note revisits inference latency specifically from a research reporting perspective — how latency should be measured and presented rigorously in papers, complementing the production-focused coverage in Inference Latency.
Why Rigorous Latency Reporting Matters in Research
When a paper claims a new method is "faster" than prior work, that claim needs the same rigor as any other research result — a fair, controlled comparison, on comparable hardware, measured correctly. Sloppy or selectively favorable latency reporting can just as easily mislead readers as sloppy accuracy reporting.
What Rigorous Latency Reporting Requires
| Requirement | Why It's Needed |
|---|---|
| Identical hardware for all compared methods | Latency is highly hardware-dependent — comparing Method A on a newer GPU against Method B on an older one is not a fair comparison |
| Report both mean and variance across multiple runs | Individual latency measurements have noise, similar to the reasoning in Statistical Significance |
| Specify batch size and sequence length used | Latency depends heavily on these — a single number without this context is not fully interpretable or reproducible |
| Include warm-up runs before measuring | Excludes one-time initialization overhead (e.g. CUDA kernel compilation) that would otherwise distort the measurement |
Code — A Rigorous Research-Grade Latency Benchmark
import time
import numpy as np
import torch
def benchmark_latency(model, input_tensor, num_warmup=10, num_trials=100):
model.eval()
with torch.no_grad():
for _ in range(num_warmup):
_ = model(input_tensor)
torch.cuda.synchronize()
latencies = []
for _ in range(num_trials):
start = time.perf_counter()
_ = model(input_tensor)
torch.cuda.synchronize()
latencies.append((time.perf_counter() - start) * 1000)
return {
"mean_ms": np.mean(latencies),
"std_ms": np.std(latencies),
"p50_ms": np.percentile(latencies, 50),
"p99_ms": np.percentile(latencies, 99),
"hardware": torch.cuda.get_device_name(0),
"batch_size": input_tensor.shape[0],
}
results = benchmark_latency(model, dummy_input)
print(results) # a complete, reproducible latency reportReporting Latency Fairly Relative to Accuracy
A method that's faster but meaningfully less accurate isn't automatically "better" — rigorous research typically reports latency alongside accuracy at multiple operating points, or explicitly acknowledges the specific accuracy-latency tradeoff being made, rather than presenting a speed improvement in isolation as unambiguously positive without this essential context.
Common Mistakes
- Comparing latency measured on different hardware configurations for different methods — this produces a fundamentally unfair, non-comparable result, regardless of how the numbers are presented.
- Reporting a single latency number without specifying batch size, sequence length, or hardware — this makes the result difficult to interpret correctly or reproduce independently.
Interview Relevance
Q: "What would you check before trusting a paper's claim that its new method is '3x faster' than a prior approach?" Whether both methods were measured on identical hardware (a fundamental requirement for a fair comparison), whether the comparison used the same batch size and input configuration for both, whether warm-up runs were excluded from timing, and whether the reported number reflects an average across multiple trials with reported variance rather than a single, potentially unrepresentative measurement. A "3x faster" claim without this context and rigor should be treated with appropriate skepticism until these details can be verified.
Practice Question
Why is reporting latency alongside accuracy, rather than in isolation, important when comparing two methods with different speed-accuracy tradeoffs?
