Orchestration: Scheduling, Placement, and Keeping the Line Full

A factory with excellent machine tools and a fast conveyor still produces nothing without a line controller — the system that decides what runs where, when, and in what order. In the AI factory that controller is the orchestration layer: the schedulers, placement logic, and job managers that assign work to silicon. This is the least glamorous part of the architecture and the one that most directly sets utilization, which — as established — is the master lever on both tokens-per-dollar and tokens-per-watt. Orchestration is where a balanced, well-built line either runs full or runs half-empty.

Start with the core job: placement. A large training run needs thousands of accelerators that can communicate efficiently, which means they should be physically close in the fabric hierarchy — ideally packed so the heaviest collective traffic stays inside the fast scale-up domains and travels the shortest paths across the scale-out fabric. Topology-aware placement is the scheduler's understanding of the building's wiring, used to assign a job's chips so that the ones which must talk most sit closest. Place the same job's chips scattered across distant racks and the collective operations crawl across congested long paths; place them in a tight, topology-aware block and the same job runs markedly faster on the same hardware. The scheduler's map of the fabric is therefore a direct input to throughput, not an administrative detail.

Placement runs into fragmentation, the orchestrator's version of a cluttered factory floor. As jobs start and finish, they leave holes — pockets of free accelerators scattered around the building. A large new job needs a big contiguous, well-connected block, but the free capacity is broken into fragments too small or too poorly connected to use. The chips are idle and available, yet the job cannot be placed efficiently. Depreciation runs on those idle chips regardless. Good orchestration fights fragmentation continuously — packing jobs to leave large usable blocks free, and sometimes migrating running work to defragment the floor — because scattered idle silicon is capacity you own and cannot sell as tokens.

Then comes the harder problem: mixing workloads with different shapes. Training jobs are large, long-running, and latency-tolerant — they want as many chips as possible for hours or weeks and do not care about millisecond response. Inference jobs are the opposite — small, latency-sensitive, and bursty, needing an answer in milliseconds and arriving in unpredictable waves. A building that serves only one type wastes capacity; a building that mixes them well keeps utilization high by filling the gaps that any single workload leaves. The orchestrator's art is to pack these together — using latency-tolerant training to soak up capacity that bursty inference is not currently using, while protecting inference's response-time guarantees — so the line stays full without breaking its commitments. This is classic production scheduling: interleave big steady jobs with small urgent ones so no station idles and no deadline slips.

Preemption and prioritization are the tools that make the mix work. When an inference burst arrives, the orchestrator may pause or shrink a training job to free chips instantly, then return them when the burst passes. When capacity is scarce, it enforces priorities so the most valuable work runs first. Done well, this is invisible and utilization stays high; done badly, it thrashes — constantly interrupting jobs so that the overhead of stopping and restarting eats the capacity it was meant to reclaim. The discipline is to keep the line full while spending as little as possible on the switching itself, exactly as a factory minimizes changeover time between production runs.

For the factory frame, orchestration is the line controller that turns a pile of balanced, well-connected silicon into a running production line. The best chips, the densest racks, and the fastest fabric produce nothing while idle, and keeping them un-idle across a churning mix of jobs, failures, and priorities is a genuinely hard scheduling problem — one that separates buildings running near their potential from those running half-empty on identical hardware. For the operator it is the daily fight for utilization; for the investor it is a decisive and often overlooked differentiator between two clusters with the same spec sheet; for the policymaker it is part of why efficiency is a systems property, not a chip property. The next lesson turns orchestration's goal into numbers — the utilization metrics that tell you, honestly, how much of the factory you are actually using.