Deployment interview questions covering serialization, serving, batch vs real-time inference, and production concerns — with fully explained answers.
Q1. What's the difference between TorchScript and ONNX?
TorchScript converts a PyTorch model into a serialized, self-contained format that can run independently of a Python interpreter, primarily within the PyTorch/LibTorch ecosystem (e.g. for C++ deployment). ONNX is a framework-agnostic format — a model exported to ONNX can run on many different runtimes and hardware backends (ONNX Runtime, TensorRT), not just PyTorch-related ones, and often benefits from significant inference speed optimizations these specialized runtimes apply. TorchScript is the natural choice for staying within the PyTorch ecosystem; ONNX is preferred for cross-framework portability or targeting specialized inference hardware.
Q2. Why is loading a model once at startup better than loading it per-request?
Loading model weights from disk into memory takes real, non-trivial time. Doing this on every incoming request adds that loading latency to every single prediction, dramatically slowing the API and wasting compute repeatedly loading identical weights. Loading once at application startup and keeping the model resident in memory for the service's lifetime means each individual request only pays the (much smaller) cost of the forward pass itself.
Q3. What's the difference between batch and real-time inference, and how do you decide which to use?
Batch inference processes a large accumulated set of inputs together, on a schedule, when no individual response is being waited on in real time (e.g. a nightly recommendation refresh). Real-time inference responds to individual requests immediately as they arrive, required whenever a live user or system is directly waiting on the result (e.g. a chatbot response). The deciding factor is simply whether the use case genuinely requires an immediate response — if not, batch inference is usually simpler, cheaper, and can use larger, more efficient batch sizes than real-time serving typically allows.
Q4. Why is unpickling a model file from an untrusted source a security risk?
Python's pickle format (which torch.save() uses by default) can encode instructions that execute during deserialization, not just passive data — a maliciously crafted pickle file can run arbitrary code on the machine that loads it. This makes unpickling untrusted data a genuine attack vector, which is why safer alternatives exist for handling models from unverified sources: torch.load(..., weights_only=True), or dedicated formats like safetensors designed specifically to exclude any code-execution capability.
Q5. What does quantization trade off, and why might a team accept the tradeoff?
Quantization reduces the numerical precision of a model's weights (e.g. from 32-bit floating point to 8-bit integers), which shrinks the model's memory footprint and typically speeds up inference, particularly on hardware with optimized low-precision support. The tradeoff is a usually small, but real and task-dependent, drop in accuracy — teams accept this tradeoff when the speed/memory benefits (faster response times, lower serving cost, ability to fit larger models within memory constraints) outweigh a measured, acceptable accuracy cost, which should always be verified directly rather than assumed negligible.
Q6. How would you diagnose whether a slow inference API is bottlenecked by the model itself or by something else?
Profile latency at the component level — preprocessing, the model's forward pass, postprocessing, and network/queueing time — rather than relying on total end-to-end latency alone. If the model's forward pass is fast but total latency is high, the bottleneck is elsewhere (inefficient preprocessing, insufficient serving capacity causing queueing under load, or network overhead) — optimizing the model itself (e.g. via quantization) wouldn't meaningfully help in that case, so identifying the actual bottleneck first is essential before investing optimization effort.
Q7. What's the purpose of a health check endpoint in a deployed model API?
A health check endpoint lets orchestration systems (like Kubernetes) and load balancers verify that a running instance is actually able to serve traffic correctly, not just that its process is technically alive. A good health check goes beyond confirming the web server responds — ideally it verifies the model is loaded and can complete a trivial forward pass successfully — so that a broken instance (model failed to load, corrupted state) can be detected and automatically removed from rotation or restarted, rather than continuing to receive and fail requests.
Q8. Why does GPU utilization matter, and what does low utilization usually indicate?
GPU utilization measures how effectively an (expensive) GPU's compute capacity is actually being used during training or serving. Low utilization usually indicates the GPU is spending significant time idle, waiting on something else in the pipeline — most commonly data loading/preprocessing on the CPU keeping up with a much faster GPU. Diagnosing and fixing the actual bottleneck (e.g. adding more DataLoader workers) is the appropriate response — simply upgrading to a more powerful GPU wouldn't meaningfully help if the GPU is already sitting idle much of the time waiting on something unrelated to its own compute speed.
