Model Serving & Inference

Model serving is the bridge between data science and production systems. A trained model is a static artifact -- a set of weights and a computation graph -- but serving it at production scale requires solving a distinct set of systems engineering problems: how do you maximize GPU utilization when requests arrive at irregular intervals? How do you serve a 70B-parameter model that does not fit on a single GPU? How do you roll out a new model version without downtime or accuracy regressions?

The fundamental tension in model serving is between latency and throughput. A GPU achieves peak FLOPS only when operating on large tensors, which means batching multiple requests together. But batching increases latency for individual requests because each request must wait for the batch to fill. Naive batching (wait for N requests or T milliseconds) wastes GPU cycles on small batches during low traffic and adds unacceptable latency during spikes. Continuous batching (also called iteration-level or inflight batching), pioneered by Orca and adopted by vLLM and TGI, solves this by inserting new requests into an active batch at each decode step of autoregressive generation, achieving 2-4x higher throughput than static batching.

For LLM inference specifically, the KV-cache is the dominant memory bottleneck. Each token in the context window requires storing key and value tensors for every attention layer. A 70B model with a 4K context window consumes ~2 GB of KV-cache per concurrent request. PagedAttention (vLLM) manages KV-cache memory like an OS manages virtual memory -- using non-contiguous pages to eliminate fragmentation -- and improves serving throughput by 2-4x compared to naive contiguous allocation.

Multi-GPU and multi-node serving uses tensor parallelism (split a single layer across GPUs) and pipeline parallelism (split layers across GPUs in stages). Tensor parallelism reduces per-token latency because all GPUs compute simultaneously, but requires high-bandwidth interconnect (NVLink at 900 GB/s, not PCIe at 64 GB/s). Pipeline parallelism tolerates lower bandwidth but increases latency for individual requests. At the fleet level, a serving system must also handle model routing (send requests to the right model version), autoscaling (scale GPU instances based on queue depth, not CPU), and graceful draining (finish in-flight requests before shutting down a replica).

An e-commerce site deploys a product recommendation model. During off-peak hours, requests trickle in at 10 RPS; during flash sales, traffic spikes to 5,000 RPS. With static batching (batch size 32, timeout 50ms), off-peak requests wait 50ms for a batch that never fills, and peak traffic queues behind full batches. Switching to continuous batching, the system processes requests individually during off-peak (no wait) and dynamically forms large batches during peak (high GPU utilization). P50 latency drops from 60ms to 15ms off-peak, and peak throughput increases 3x without adding GPUs.