The Economics of a Token: Prefill, Decode, and Cost per Token

To schedule inference economically you must first price a token correctly, and the key fact is that not all tokens are equal. Input tokens and output tokens are produced by different computational phases with different cost structures. Input tokens are consumed during prefill, a single parallel pass over the whole prompt that saturates the accelerator's compute. Thousands of input tokens are processed together in one efficient burst, so the cost per input token is low. Output tokens are produced during decode, one token per pass over the entire weight set, a serial and memory-bound process. Each output token drags the full model through memory again. This is why serving platforms price output tokens several times higher than input tokens: the underlying cost genuinely differs.

This asymmetry drives the first economic rule of scheduling: output length is the primary cost lever. A request that reads a long document and returns a short answer is cheap; a request that returns a long answer is expensive, almost regardless of input size. Operators cap and control output length, use models that are concise, and design products that do not generate more tokens than the user needs. A verbose default is a direct tax on margin. For an investor reading a company's unit economics, the ratio of output to input tokens per request is a leading indicator of gross margin, more informative than headline request volume.

The second economic fact is that cached tokens are nearly free. When prefix caching lets the engine skip the prefill over a shared prompt, those input tokens cost almost nothing because the work was done once and amortized across every reuse. This changes prompt design into cost engineering. Stable, shared content, a system prompt, a document, a conversation history, placed at the front of the prompt becomes cacheable and effectively free on repeat. Variable content placed at the front breaks the cache and forces full prefill every time. The same conversation, structured two different ways, can differ in cost by a large factor. Token economics is therefore partly a matter of prompt architecture, and the serving layer rewards prompts that expose reuse.

The third fact is that batching sets the unit cost of decode. Because decode is memory-bound, the accelerator reads the full weight set from memory once per step regardless of how many requests share that step. Serve one request per step and that memory read is amortized over one token; serve fifty and it is amortized over fifty. Larger batches therefore cut the cost per output token, up to the point where memory for the key-value cache runs out or latency targets break. This is the central trade the scheduler manages: throughput and unit cost pull toward larger batches, while latency pulls toward smaller ones. The optimal batch size is not a constant; it moves with traffic, with the mix of request lengths, and with the latency the product promises.

Put these together and cost per token becomes a function the scheduler actively optimizes rather than a fixed price. Blended cost per token falls when the scheduler batches decode aggressively, caches to eliminate redundant prefill, routes easy work to cheap models and old hardware, and caps runaway generations. It rises when requests arrive with unique prefixes that defeat caching, demand long outputs, or carry latency targets so tight that batches must stay small. A sophisticated operator instruments all of this: cost per token by product, cache hit rate, mean output length, batch occupancy. These are the real financial metrics of an inference business, and they are set in the serving layer.

For a policy reader, the token-economics view clarifies where efficiency and therefore energy use are actually determined. Two providers running the same model on the same hardware can differ severalfold in energy per useful answer purely through scheduling: caching, batching, and routing. Efficiency mandates that target the chip miss most of the picture, because the software layer that schedules inference against power and hardware is where the majority of the efficiency is won or lost. The token is the unit of account, and the scheduler is the accountant that decides how many accelerator-seconds and how many joules each token costs.