Peak Versus Sustained: The Utilization Gap

Peak performance and sustained performance are two different quantities, and confusing them is the most common and most expensive mistake in this field. Peak is what the arithmetic units can do if every one of them is busy every cycle with the ideal data already in place. Sustained is what actually happens once you account for the numbers arriving late, the units sitting idle, and the overhead of coordinating everything. The ratio between the two — call it utilization — routinely lands well below the ceiling, and the shortfall is not waste to be scolded away. It is structural.

Consider why a unit goes idle. To multiply two numbers it must first have both numbers in a register beside it. Those numbers came from on-package memory, which was filled from host memory, which was filled from storage. If any stage in that chain is slower than the arithmetic unit consumes, the unit stalls and waits. Since arithmetic has grown far faster than the speed of moving data, the stall is the normal case. The chip spends a large share of its cycles waiting for data it has already been told to compute on. Adding more arithmetic units does not fix this; it makes each one wait longer relative to its capacity.

A worked illustration makes the stakes concrete. Suppose an accelerator advertises a headline throughput and a workload realizes forty percent of it — a plausible figure for a demanding job on a well-built system. A buyer who priced the fleet against the headline number has over-bought by more than half. Push utilization from forty to fifty percent through better data staging and scheduling, and you have found a quarter more capacity without buying a single additional chip. That is the highest-return work in the building, and none of it touches the silicon. It touches everything around the silicon.

Utilization is also workload-dependent, which is why a single benchmark number misleads. Training a large model runs long, dense, predictable computations that keep units busy — utilization tends to be higher. Serving that model to users runs short, spiky, latency-sensitive computations where the chip often waits for the next request or processes tiny batches that cannot fill the arithmetic array — utilization tends to be lower. The same chip therefore posts very different realized performance depending on the job. An operator running inference and an operator running training can own identical hardware and extract wildly different value from it.

This is where orchestration enters as the decisive discipline. The software layer that decides which work runs where, batches requests together to fill the arithmetic array, overlaps data movement with computation so units never wait, and packs jobs onto chips to avoid stranded capacity is the primary lever on sustained performance. Two operators with the same silicon and the same memory can differ by a large margin in delivered work purely because one schedules and batches well and the other does not. The chip is fixed; the utilization is earned.

There is a measurement discipline that follows. Operators who manage this well instrument the realized fraction directly — how busy the arithmetic units actually are, how often they stall, how much memory bandwidth is consumed versus available. They treat a low realized fraction as a defect to hunt down, tracing it to a starved memory channel, an unbalanced schedule, or a batch too small to fill the array. Operators who manage it poorly quote the datasheet and wonder why the bill is so high.

For the investor, sustained-versus-peak is a diligence question with a precise form: not "how fast is the chip" but "what fraction of the chip does this operator actually harvest, and is that fraction improving." A fleet running at a low realized fraction is carrying enormous latent capacity that better software could unlock — either an opportunity or a warning, depending on whether the team can capture it. For the policy reader, it explains why counting installed silicon overstates real capability: a large fleet run at low utilization delivers far less than its headline suggests, and a smaller fleet run well can out-produce it. Capability is realized work, not installed peak, and the multiplier between them is the system.