Token-Economics-Aware Scheduling
A token is a unit of cost, and not all tokens cost the same. Input tokens are cheap because prefill is parallel; output tokens are expensive because decode is serial and memory-bound. Cached tokens are nearly free. Once you price tokens correctly, scheduling becomes an economic optimization, not just a systems one. The scheduler is spending a fixed budget of accelerator-seconds and power, and every admission decision allocates that budget. Batch to lower unit cost, cache to eliminate work, route to cheap resources, and schedule against real-time power prices. The operators who treat inference as a cost-accounting problem, with the scheduler as the accountant, win on margin. The rest pay retail for every token.
Knowledge check
Why do output tokens cost more than input tokens, and how does that shape scheduling?
Show the answer
Input tokens are processed in a single parallel prefill pass, so they are cheap per token. Output tokens are generated one at a time in the serial, memory-bound decode phase, each requiring a full pass over the weights, so they are far more expensive. Scheduling therefore prioritizes packing decode work efficiently, capping output length, batching aggressively, and caching to avoid redundant prefill, because decode is the dominant cost driver.