The Key-Value Cache: Memory Reuse, Paging, and Prefix Sharing

Decode is only tractable because of the key-value cache. To generate each new token, the model needs the attention keys and values for every previous token in the sequence. Recomputing them from scratch at every step would make generation quadratic and hopeless. Instead the engine computes each token's keys and values once and stores them. That store is the key-value cache, and it is the dominant consumer of accelerator memory in production serving. Understanding it explains most of what serving infrastructure actually does.

The cache grows linearly with sequence length and with the number of concurrent requests. A single long conversation can hold gigabytes of cache. Multiply by hundreds of concurrent users and the cache, not the model weights, becomes the binding constraint on how many requests one accelerator can serve. This reframes the whole problem. Serving capacity is a memory-management problem. The accelerator has fixed high-bandwidth memory; the weights take a fixed slice; everything left over is the budget for concurrent requests, and the cache is how you spend it.

Early engines allocated cache memory contiguously, reserving a block sized for each request's maximum possible length. This wasted enormous amounts of memory. A request that might generate two thousand tokens but actually generates fifty still reserved the full block. Fragmentation compounded the loss. The breakthrough was paged attention, which borrows the idea of virtual memory from operating systems. The cache is divided into small fixed -size blocks, or pages, allocated on demand as a sequence grows. A sequence's pages need not be contiguous; a lookup table maps logical position to physical page. Waste drops from roughly sixty or seventy percent to a few percent, and the freed memory converts directly into more concurrent requests and higher throughput.

Paging unlocks the second big win: reuse. Because the cache is now addressable in blocks, identical blocks can be shared across requests. This is prefix caching, and it is where token economics gets interesting. Many requests share a common prefix. A system prompt of a few thousand tokens is prepended to every user query. A long document is queried repeatedly in a retrieval application. A multi-turn conversation resends its entire history on each turn. In every case the prefix is identical across requests or across turns. With prefix caching the engine computes that prefix's keys and values once, stores the blocks, and reuses them for every request that shares the prefix. The expensive prefill over the shared prefix happens a single time instead of thousands of times.

The savings are structural, not marginal. If a system prompt is four thousand tokens and a user question is fifty, prefix caching removes the four thousand tokens of prefill from nearly every request. Time-to-first-token drops sharply because the model resumes from cached state instead of reprocessing the prefix. Cost per request falls in proportion. This is why serving platforms expose prefix caching as a headline feature and why prompt structure now affects unit economics: putting the stable, shared content at the front of the prompt makes it cacheable, while interleaving variable content defeats the cache. The prompt is now a performance surface.

Reuse extends beyond a single accelerator. In large deployments the cache becomes a tiered storage system. Hot blocks stay in accelerator memory. Warm blocks spill to host memory or local storage and are pulled back when a matching request arrives. Cross-request and cross-node cache sharing lets a whole cluster behave as one large cache, so a document processed by one node primes the cache for the same document hitting another node. This is real infrastructure engineering: eviction policies, hashing prompts to cache keys, deciding what to keep and what to recompute, and doing it fast enough that lookup is cheaper than regeneration.

The cache also shapes hardware choice. Because concurrency is bounded by memory, an accelerator with more high-bandwidth memory serves more simultaneous requests, and memory bandwidth, not peak compute, often sets decode speed. Operators evaluating chips for inference weigh memory capacity and bandwidth as heavily as raw arithmetic throughput, because the cache is the bottleneck the workload actually hits. The takeaway is blunt: the key-value cache is the beating heart of the serving stack. Manage it well with paging and prefix reuse and one cluster serves several times the traffic. Manage it badly and you pay for idle compute stranded behind a memory wall.