This closing note of the Deployment category covers model optimization techniques applied specifically to make a trained model faster and lighter for production inference — distinct from optimization during training.
Common Deployment-Time Optimization Techniques
| Technique | What It Does | Typical Tradeoff |
|---|---|---|
| Quantization | Reduces numerical precision of weights (e.g. FP32 → INT8) | Smaller model, faster inference; small accuracy drop, usually minor |
| Pruning | Removes weights or entire structures with minimal contribution to output | Smaller, sometimes faster model; requires care to avoid meaningful accuracy loss |
| Knowledge distillation | Trains a smaller "student" model to mimic a larger "teacher" model's behavior | Much smaller/faster model; typically a larger accuracy tradeoff than quantization/pruning |
| Graph/operator fusion | Combines multiple sequential operations into a single, more efficient fused operation | Faster inference with no accuracy cost — a "free" optimization when applicable |
| Compilation (e.g. TensorRT, torch.compile) | Compiles the model graph into highly optimized, hardware-specific code | Faster inference with typically minimal or no accuracy cost |
Code — Post-Training Quantization
import torch
model.eval()
quantized_model = torch.quantization.quantize_dynamic(
model,
{torch.nn.Linear}, # which layer types to quantize
dtype=torch.qint8
)
# The quantized model is smaller and typically faster on CPU inference,
# with a usually small, task-dependent drop in accuracy -- always measure it directlyCode — Using torch.compile for Faster Inference
model = MyModelClass()
model.load_state_dict(torch.load("model_weights.pt"))
model.eval()
compiled_model = torch.compile(model) # compiles the model graph for faster execution
with torch.no_grad():
output = compiled_model(input_tensor)
# The first call includes compilation overhead; subsequent calls benefit from the speedupThe Right Order of Operations
Optimization should always follow, not precede, correctness — first get a working, accurately-evaluated model (per Model Evaluation), then apply optimization techniques, then re-evaluate the optimized model's accuracy directly, since some techniques (quantization, pruning, distillation) do trade off some accuracy, and this tradeoff needs to be measured and consciously accepted, not assumed to be negligible.
Common Mistakes
- Applying aggressive optimization (heavy quantization, aggressive pruning) without re-measuring accuracy afterward — assuming the accuracy impact is negligible without verifying it directly can silently ship a meaningfully degraded model.
- Optimizing a model before its architecture and training are finalized — optimization effort spent on a model that later changes significantly is wasted; optimize the final, validated model, not an intermediate one.
Interview Relevance
Q: "Why is it important to re-evaluate a model's accuracy after applying deployment-time optimizations like quantization or pruning, rather than assuming the impact is negligible?" Techniques like quantization and pruning deliberately trade off some model precision or capacity for speed and size benefits — the actual accuracy impact varies by model, task, and how aggressively the technique is applied, and can sometimes be more significant than expected. Re-evaluating on the same held-out test set used for the original model ensures this tradeoff is measured and consciously accepted, rather than silently shipping a meaningfully worse model under the assumption that optimization is "free."
Key Takeaways — Deployment
- Serialization (native format, TorchScript, or ONNX) converts a trained model into a portable artifact — the right format depends on the target deployment environment's language and runtime needs.
- Never unpickle model files from untrusted sources — this is a genuine security risk, not just a data format concern.
- A model-serving API (e.g. via FastAPI) needs input validation, structured error handling, and health checks to be production-ready, beyond a minimal working endpoint.
- Docker containerizes the serving application and its exact dependencies for reproducible deployment across environments.
- Batch vs real-time inference is a foundational early decision shaping the entire serving architecture; GPU usage and optimization techniques should be justified by measured need, not assumed by default.
Next: Production DL & MLOps covers what happens after a model is deployed — monitoring, drift detection, experiment tracking, and the ongoing operational practices that keep a model reliable over time.
Practice Question
Why should model optimization techniques like quantization always be applied after, not before, finalizing a model's architecture and training?
