Most production deep learning models are deployed on cloud infrastructure rather than self-managed physical servers — this note covers the common cloud deployment patterns and the tradeoffs between them.
Common Cloud Deployment Patterns
| Pattern | Description | Best For |
|---|---|---|
| Managed model-serving platforms | Cloud-provider-specific services purpose-built for ML model hosting (e.g. SageMaker, Vertex AI) | Teams wanting to minimize infrastructure management overhead |
| Container orchestration (Kubernetes) | Deploying Docker containers onto a managed or self-run Kubernetes cluster | Teams needing fine-grained control, or already using Kubernetes for other services |
| Serverless functions | Deploying inference code as a function that runs on-demand, scaling to zero when idle | Low, sporadic traffic where paying only for actual usage matters |
| Self-managed virtual machines | Directly running the serving application on provisioned VM instances | Maximum control, but the most operational overhead |
Key Tradeoffs to Weigh
- Management overhead vs control: managed platforms reduce operational burden but offer less flexibility; self-managed infrastructure offers full control at the cost of more engineering effort to build and maintain.
- Cost model: always-on instances have predictable cost but pay even during idle periods; serverless/auto-scaling options can be more cost-efficient for variable traffic but may introduce cold-start latency.
- Scaling behavior: auto-scaling handles traffic spikes without manual intervention, but must be configured thoughtfully — under-provisioned scaling limits can still cause outages during genuine spikes.
Cold Start — A Real Serverless Consideration
Serverless deployments that scale to zero when idle must "cold start" — provision resources and load the model into memory — when the first request arrives after a period of inactivity, adding meaningful latency to that first request. For latency-sensitive applications, this can be a genuine problem; mitigations include keeping a minimum number of instances warm at all times, at the cost of losing some of serverless's cost efficiency.
A Simple Decision Guide
if traffic_is_low_and_sporadic and cold_start_latency_is_acceptable:
choice = "serverless functions"
elif need_fine_grained_control or already_using_kubernetes_elsewhere:
choice = "Kubernetes"
elif want_minimal_operational_overhead and standard_use_case:
choice = "managed model-serving platform"
else:
choice = "evaluate based on team's specific operational capacity and constraints"Common Mistakes
- Defaulting to the most operationally heavy option (self-managed Kubernetes) for a simple use case that a managed platform or serverless function would handle perfectly well with far less engineering overhead.
- Choosing serverless for a latency-critical, high-traffic application without accounting for cold-start latency, or without configuring a minimum warm instance count to avoid it.
Interview Relevance
Q: "What is 'cold start' in the context of serverless model deployment, and why might it matter for a latency-sensitive application?" A serverless deployment that scales to zero during idle periods must provision compute resources and load the model into memory when the first request arrives after inactivity — this initialization adds meaningful latency to that first request, which can be unacceptable for latency-sensitive, real-time applications. Mitigations include keeping a minimum number of "warm" instances always running, trading away some of serverless's cost efficiency for consistently lower latency.
Practice Question
A model serves highly variable traffic — near zero requests overnight, and large spikes during business hours. What deployment pattern would you consider, and what tradeoff would you need to manage?
