From Node to Cluster: Networking and Scale
A cluster is thousands of chips pretending to be one, and the pretense holds only as well as the fabric and the software that stitch them together. The collective operation — where every chip waits on the slowest path before any can proceed — makes network topology and parallelism strategy first-order determinants of throughput, and the overlap of computation with communication is a software achievement, not a silicon one. At scale, failure is constant, so the honest metric is goodput: the fraction of time that produces kept work. Checkpointing, fast recovery, and straggler eviction — none of them properties of the die — decide it. Two operators with identical hardware can deliver very different finished work. The cluster's capability lives in the whole system.
Knowledge check
What is a straggler in a large distributed job, and why can a single slightly-slow chip drag down the effective speed of the entire cluster?
Show the answer
A straggler is one chip running slightly slower than the rest — throttled by heat, a marginal link, or an unlucky share of work. Because the collective operation is a synchronization barrier that waits on its slowest participant, every other chip idles at that barrier until the straggler finishes. So one degraded component out of thousands pulls the whole job's effective speed toward its own, even though every component still reports as working.