The Line Controller: Orchestration, Utilization, and Throughput Thinking
Excellent machine tools and a fast conveyor still produce nothing without a line controller. Orchestration — placement, defragmentation, and the interleaving of steady training with bursty inference — is what keeps expensive silicon un-idle, and utilization is the master lever on both cost and energy per token. Then measure honestly. Model FLOPs utilization exposes that even great buildings convert only about half their theoretical arithmetic into progress; goodput separates useful output from rework; failure domains keep certain-to-occur faults from idling the whole line. The building is a factory, the product is tokens, and every component is judged by one question: does it move realized tokens-per-dollar and tokens-per-watt upward at sustained load?
Knowledge check
What is the difference between raw throughput and goodput, and why does the distinction matter for judging an AI factory?
Show the answer
Raw throughput counts all work the hardware performs, including work later discarded; goodput counts only work that advanced the finished product. Large runs waste effort — failed jobs restart from checkpoints, stragglers force reruns, corrupted steps are redone — so a cluster can look busy while much of its motion is rework. Goodput is the factory's yield: good units shipped, not units produced. Judging a building by goodput (and by MFU and failure behavior) reveals its real output at sustained load, which chip counts and peak specifications hide.