The Inference Serving Stack
Training builds the model once. Serving runs it a billion times. The economics live in the serving layer, and serving is a scheduling problem before it is a hardware problem. A modern serving engine is not a wrapper around a model file; it is a real-time scheduler that packs requests, reuses memory, and keeps expensive accelerators saturated. Continuous batching, key-value cache reuse, and paged memory are the three levers that turn idle silicon into throughput. Get the software layer right and one cluster does the work of three. Get it wrong and you pay for accelerators that spend most of their life waiting. The winners optimize the scheduler, not just the chip.
Knowledge check
Why is high-throughput inference described as a scheduling problem rather than a raw compute problem?
Show the answer
Because accelerator utilization is set by how well the serving engine packs many concurrent requests, reuses cached memory, and overlaps work. The same chip delivers wildly different throughput depending on batching and memory management, so the scheduler, not the raw FLOPs, determines cost per token.