The Cost of an Idle Accelerator

Start with the balance sheet. An accelerator is a capital asset. It is bought once and depreciates on a schedule regardless of whether it computes. Power, cooling, networking and floor space are provisioned for it whether it is busy or idle. So the cost of a unit of useful computation is roughly the fixed cost of the fleet divided by the useful work it actually performs. The numerator is set on purchase day. Only the denominator moves.

That makes utilization the central economic variable of an AI factory. Every idle hour on a graphics processing unit, or GPU, is fixed cost with no output against it. This is not new. Airlines, steel mills and semiconductor fabs all live by the same arithmetic. What is new is the size of the fixed cost per rack. NVIDIA's rack-scale systems, such as the GB200 NVL72 that Vertiv designed a reference power and cooling architecture for in October 2024, concentrate a great deal of capital into a single cabinet. The denser the capital, the more each idle minute costs.

Separate three measures that are often blurred. Allocation is the share of accelerators assigned to some job. Utilization is the share of their capacity actually doing arithmetic. Goodput is the share of that arithmetic that advances a result and is not later thrown away. A cluster can be fully allocated and poorly utilized, because jobs are waiting on data or on each other. It can be well utilized and produce poor goodput, because work is repeatedly lost to failures and redone. Vendors usually quote the first. Owners care about the third.

The scheduler controls allocation directly. Its tools are the queue, priority and packing. A good scheduler keeps the queue full so that freed nodes are claimed immediately. It sets priorities so that the most valuable work is not stuck behind experiments. It packs jobs so that small jobs fill the gaps left by large ones. Slurm's backfill scheduling, which runs smaller jobs in the holes before a large reserved job starts, is a classic example of this discipline.

Utilization depends on the workload itself and on the infrastructure beneath it. Large training jobs are synchronous. Thousands of GPUs compute a step, then stop and exchange results before the next step begins. If one device is slow, every other device waits for it. Meta's paper on Llama 3, published in July 2024, states the consequence plainly: the synchronous nature of training makes it less fault-tolerant, and a single GPU failure may require a restart of the entire job. Straggling has the same shape. One throttled chip taxes the whole cluster.

This is why the control plane matters economically, not just operationally. Consider three ordinary events. A chip runs hot because a cooling loop is marginal, so it lowers its clock speed and becomes a straggler. A memory error takes a node offline, so the job must stop and restart from its last saved state. A power limit forces the facility to cap consumption, so every device slows at once. None of these shows up as low allocation. All of them show up as lost goodput. Each one crosses the border between facility and IT.

The Llama 3 paper also reported something subtler. Meta observed a daily variation of one to two percent in training throughput, which it attributed to higher midday temperatures affecting the chips' dynamic voltage and frequency scaling. That is the building, reaching directly into the output of the model. Weather became a line item.

Kubernetes has moved to give schedulers finer control over the devices themselves. Its Dynamic Resource Allocation feature, generally available since version 1.34 in 2025, lets a job request a specific kind of accelerator rather than a generic count. Finer requests allow tighter packing, and tighter packing raises utilization. It is an unglamorous improvement with direct economic effect.

What should a newcomer take from this? That the useful question about any AI factory is not how many accelerators it has, but how much useful work those accelerators deliver per day, and what stands between the two numbers. The gap between installed capacity and goodput is where the control plane earns or loses its keep.

Faraday's read is that utilization is the most under-reported number in AI infrastructure. Capacity figures are announced. Megawatts are announced. Chip counts are announced. The share of that capacity doing useful work is almost never published, and it is the figure that decides whether the capital earns a return. When an operator does report it, pay attention. When it declines to, ask why.