Scheduling Against Power and Price: SLO-Aware and Power-Aware Serving

Once tokens are priced, the scheduler's job is to spend a fixed budget of accelerator-seconds and electricity as efficiently as possible while meeting the promises made to users. Those promises are service-level objectives: a target time-to-first-token, a target inter-token latency, a target completion time for batch jobs. SLO-aware scheduling treats these targets as constraints and maximizes throughput within them. The scheduler admits latency-critical requests immediately and packs latency-tolerant requests into the gaps, filling the accelerator without breaking any promise. This is classic real-time scheduling applied to inference: hard deadlines for interactive work, best-effort throughput for the rest, all on the same fleet.

The mechanism is prioritization plus preemption. Interactive requests get high priority and are protected from large prefill bursts through chunked prefill, so their token stream stays smooth. Background requests get low priority and are admitted only when spare capacity exists; they can be paused, their cache spilled to slower memory, and resumed later when the interactive load ebbs. Because inference demand is bursty and follows daily human rhythms, there is substantial idle capacity in the troughs. A scheduler that shifts flexible work into those troughs raises average utilization without adding hardware, which is the same lever as batching and routing viewed from the demand side.

Power-aware scheduling extends this idea to electricity, and it is where the serving layer meets the grid. Electricity is not a fixed cost. Wholesale prices swing hour to hour, sometimes dramatically, driven by demand and the availability of wind and solar. Carbon intensity swings with them. A scheduler that can see these signals can shift latency-tolerant work toward cheap, clean hours and away from expensive, dirty peaks. Batch jobs run overnight or midday when solar is abundant; they throttle back when prices spike. Across a fleet spanning regions, work migrates toward whichever data center currently has the cheapest, cleanest power. The inference load becomes partly dispatchable, something the grid can lean on rather than merely a rigid draw.

This dispatchability is strategically important because AI data centers are enormous, concentrated electrical loads that strain local grids and face multi-year waits for new power. A load that can flex, cutting draw during peaks and shifting flexible work in time, is far easier to site and interconnect than a load that demands constant full power. Some operators go further and co-locate with generation or negotiate demand-response arrangements, cutting inference draw when the grid is stressed in exchange for cheaper or faster power access. The scheduler is the instrument that makes this possible: it is the software that decides, second by second, how much power the facility actually pulls. Power-aware scheduling turns a data center from a price-taker into an active participant in the power market.

There is a hard limit to flexing, though, and it is set by the workload mix. Interactive inference cannot be time -shifted; a user waiting for a chat response will not accept a three-hour delay because power is cheaper tonight. Only the latency-tolerant fraction of the load is dispatchable. So the value of power-aware scheduling depends on how much of the fleet's work is flexible: heavy batch and background processing offers large flexibility, pure real-time serving offers little. The best operators deliberately cultivate a portfolio with enough flexible work to act as ballast, using it to smooth power draw, absorb price spikes, and improve utilization, while protecting the interactive core.

The synthesis across this module is that scheduling is where the three efficiency levers converge. Batching lowers unit cost. Caching eliminates redundant work. Routing places work on cheap models, cheap hardware, cheap regions, and cheap hours. All three are exercised by the scheduler, and all three ultimately trade against the same currency: accelerator-seconds and joules per useful token. An operator who instruments and optimizes this layer runs the same hardware at a fraction of the cost and energy of one who does not. For an investor, scheduling sophistication is a direct read on margin durability. For a policy reader, it is the layer where inference efficiency, and the grid impact of AI, is actually decided. The chip sets the ceiling; the scheduler sets how close you get to it.