Inference Serving & Model-Routing Infrastructure

Glossary

Continuous batching: An iteration-level scheduling technique in which the serving engine decides at every decode step which sequences run, immediately admitting a waiting request when one finishes so the batch never sits half-empty. It is the core throughput lever of modern serving engines.
Decode: The second inference phase, in which the model generates output one token at a time. Each step is serial and memory-bound, requiring a full pass over the model weights, which makes output tokens the dominant cost driver in serving.
Disaggregated serving: A serving architecture that runs the compute-bound prefill phase and the memory-bound decode phase on separate, specialized hardware pools, handing the key-value cache between them over a fast interconnect so each pool is sized to the phase it serves.
Key-value cache: The stored attention keys and values for every token already processed in a sequence, kept so decode need not recompute them each step. It grows with sequence length and concurrency and is usually the binding memory constraint on how many requests one accelerator can serve.
Model routing: A decision layer in front of a fleet of models that sends each request to the cheapest model, or hardware tier, that meets the required quality and latency. Because request difficulty is skewed, routing the easy majority to cheap resources sharply cuts blended cost.
Paged attention: A key-value cache management method that divides the cache into small fixed-size blocks allocated on demand, using a lookup table to map logical positions to physical blocks. It borrows virtual-memory ideas to cut memory waste and raise concurrency.
Prefill: The first inference phase, in which the model reads the entire prompt in one parallel, compute-bound pass and populates the key-value cache. Prefill cost scales with input length and is cheap per token because many tokens are processed together.
Prefix caching: Reusing the cached keys and values of a shared prompt prefix, such as a system prompt or document, so the expensive prefill over that prefix runs once instead of on every request. It lowers time-to-first-token and cost and makes prompt structure a performance surface.