The Memory-Bound Economics of Inference

Serving a large language model has two phases with opposite characters, and telling them apart is the key to inference economics. The first is prefill, where the model ingests the prompt. Prefill processes all input tokens together in a large batch, which gives it high arithmetic intensity: many operations per byte of weights read, because each weight read is reused across the whole batch. Prefill is often compute-bound and uses the accelerator's arithmetic units efficiently. The second phase is decode, where the model generates the response one token at a time. Each generated token requires a full pass over the model's weights, but only for that single token's worth of work, so the arithmetic intensity collapses toward one operation per byte. Decode is memory-bound, and decode is where most of the serving time and cost of a long response actually goes.

Because decode dominates and decode is memory-bound, the token generation rate is set by memory bandwidth. To emit one token you must stream the model's weights out of memory once; the time to do that is the weight size divided by the bandwidth. Multiply the parts and the rule is stark: for single-stream decode, tokens per second scales with bandwidth and is nearly indifferent to compute. This is why an accelerator with more memory bandwidth generates tokens faster on the same model, and why a chip with abundant compute but modest bandwidth disappoints on inference. The roofline from the first module has become a price sheet.

Batching is the primary lever against this, and it works by raising arithmetic intensity. If you serve many users at once, you read each weight from memory once and apply it to many tokens in flight simultaneously, amortizing the expensive memory traffic across more useful output. Larger batches move decode up the sloped part of the roofline toward the ridge point, raising throughput per unit of bandwidth and lowering cost per token. But batching has limits: it adds latency as requests wait to be grouped, it demands enough memory to hold many users' intermediate state at once, and beyond a point it saturates and stops helping. The art of efficient serving is largely the art of batching decode as aggressively as latency budgets and memory capacity allow.

That intermediate state deserves attention because it is itself a bandwidth consumer. As a model generates, it caches the intermediate representations of all prior tokens so it need not recompute them, and this cache grows with sequence length and with the number of concurrent users. It occupies precious memory near the compute, competing with the model weights for both capacity and bandwidth. Long contexts and many simultaneous users inflate this cache until it, not the weights, becomes the dominant memory pressure. Managing it, compressing it, paging it, sharing it across requests, is one of the busiest areas of serving optimization precisely because it sits on the bandwidth-and-capacity budget that determines cost.

My read: once you see decode as a bandwidth transaction, the entire menu of inference optimizations organizes itself. Quantizing weights lowers the bytes moved per token and directly raises the token rate. Sparsity and mixture-of-experts designs try to touch fewer weights per token, cutting the memory traffic per output. Speculative decoding uses a small model to draft tokens that the large model verifies in a batch, converting some memory-bound single-token steps into more intensity-rich batched ones. Every one of these techniques, however it is dressed, is an attempt to move more useful output per byte of memory traffic. They are all maneuvers on the same roofline, aimed at the same scarce resource.

And the resource itself is handed down from the GPU generation. The bandwidth you are amortizing, the memory capacity you are fitting the cache into, the ratio between them: all were set when the accelerator was designed, together with its HBM generation. When the next generation widens the memory interface, decode speeds up and cost per token falls across the board, rewarding no cleverness on your part. When memory scarcity raises the cost of each accelerator, the cost per token rises regardless of your software. The token price sits at the bottom of a causal chain whose top is the GPU generation, and the middle links, HBM design and HBM supply, are the ones this course has walked through. The final lesson turns to the architectures trying to loosen the chain.