The Fabric Is the Computer
Faraday's read: the industry counts chips because chips are easy to count. Buyers announce accelerator numbers the way navies once announced hulls. But a training cluster is not a pile of accelerators — it is one machine wired together, and a machine is only as fast as the wire between its parts. The under-tracked constraint on AI compute delivery is not how many processors you bought. It is how well they can talk. Track the fabric, and the surprises stop surprising you.
Knowledge check
A cluster's accelerators each run at full speed in isolation, yet the cluster trains far slower than their combined peak suggests. Give the structural reason and name the failure mode that causes it.
Show the answer
Large-model training is synchronous: at each step every accelerator must exchange results through a collective operation (such as all-reduce) before any can proceed. The step therefore finishes at the pace of the slowest exchange, not the average one. The failure mode is the straggler — a single slow link, congested path, or lagging node stalls the entire cluster, so aggregate compute is bounded by the tail of the communication distribution rather than the mean of the compute.