Deferrable Versus Real-Time: Building the Shiftable Queue

Temporal shifting rests on one prior decision: correctly separating work that can wait from work that cannot. Get the classification wrong and everything downstream fails, either by delaying work that had a hidden deadline or by refusing to shift work that actually had slack. So the queue is where this lesson lives.

Begin with the taxonomy of demand, not domains, but job types. Latency-critical inference is the obvious hard constraint: interactive chat, live coding assistants, real-time translation, fraud scoring in a payment flow. These carry a service-level objective in milliseconds and must run on arrival. They set the facility's non-negotiable floor of power draw. Deferrable work is everything with slack: batch embedding of a document corpus, offline evaluation and benchmarking, synthetic data generation, model distillation runs, periodic re-indexing, and most training. Between the two sits a gray band, throughput-oriented but soft-deadline work like a report due by morning, which can shift within a window but not indefinitely.

The scheduler encodes this as metadata on every job: a deadline, a priority, and a preemptibility flag. The deadline is the latest acceptable completion time. Priority breaks ties when capacity is scarce. Preemptibility states whether the job can be checkpointed and paused without corruption. Together these three fields turn an opaque workload into a set of movable objects with known constraints. A job with a deadline eight hours out and a preemptible flag is pure optionality; a job due in fifty milliseconds is a fixed cost. The scheduler's power savings come entirely from the first kind.

Checkpointing is the enabling mechanism, and it deserves respect. To pause a deferrable job and reclaim its power, the system must save enough state to resume later without losing progress. For training, that means periodic snapshots of model weights and optimizer state, which are large and expensive to write, so checkpoint frequency is itself a tuned parameter: too rare and a preemption throws away hours of work, too frequent and the snapshot overhead eats the compute you were trying to save. For batch inference, checkpointing is cheaper, often just tracking which items in the batch are done. The cost and granularity of checkpointing set the practical floor on how finely a job can be shifted.

Now the shape of opportunity. Wholesale power prices and grid carbon intensity are not flat. They follow daily and weekly cycles driven by human demand and renewable supply. Solar-heavy grids crater their prices and emissions in the midday sun and spike them in the evening ramp as the sun sets and demand peaks. Wind-heavy grids swing on weather. Overnight, demand falls and cheap baseload or surplus renewables dominate. The spread between the best and worst hour of a day is routinely large, sometimes several-fold in price and in carbon per kilowatt-hour. A deferrable job is a bet the scheduler can place on the cheap, clean hours instead of the expensive, dirty ones.

The policy that captures this is a deadline-aware queue drain. Each deferrable job has an execution window from now until its deadline. The scheduler forecasts price and carbon across that window and schedules the job into the best hours the window allows, holding it in the queue otherwise. A job due in twelve hours can wait for the overnight trough; a job due in one hour must run in the next best of a narrow set. The longer the deadline, the more of the curve the scheduler can shop across, which is why encouraging users and pipelines to declare honest, generous deadlines is one of the highest-return interventions available. Every hour of declared slack widens the search space.

Two disciplines keep this honest. First, deadlines are hard: the scheduler must guarantee completion by the stated time even in the worst forecast, which means it runs a job before the deadline regardless of price if no better hour remains. Optimization happens inside the deadline, never through it. Second, starvation control: a low-priority job repeatedly bumped by newer, higher-priority work must eventually be forced to run as its deadline approaches, or the queue quietly breaks its promises. A well-built shiftable queue is therefore not a loose backlog but a disciplined scheduler that treats time as a resource to be spent carefully, buying cheap clean power with slack it was given for free.