Scale-Up: Binding Chips Into One Machine
Faraday's read: the most important move in accelerator design over the last decade was not on the chip. It was between chips. The insight that memory-semantic bandwidth — letting one processor reach into another's memory as if it were its own — could be made cheap enough to stand in for a single monolithic processor is what makes the rack-scale machine possible. NVLink and the switch that fans it out are how you cheat the reticle limit. You cannot build a chip the size of a tray. You can wire a tray of chips until it behaves like one. That is scale-up: fabrication's ceiling, defeated by interconnect.
Knowledge check
Explain why a dedicated accelerator-to-accelerator link plus a non-blocking crossbar switch turns a group of separate processors into something that behaves like one larger processor, and what physical limit this strategy is working around.
Show the answer
A dedicated high-bandwidth link (NVLink) gives each accelerator far more bandwidth to its peers than a general-purpose bus like PCIe, and does so with memory semantics — a processor can read and write a peer's memory directly rather than copying through a host. A non-blocking crossbar (NVSwitch) connects every accelerator to every other at full bandwidth simultaneously, so any pair communicates at the same high rate with no contention. Together they make the group's combined memory and compute usable as a single pooled resource. The physical limit being worked around is the reticle limit: a single die cannot exceed the maximum size a lithography scanner can pattern, so you cannot simply build one enormous processor — you must wire many together tightly enough to substitute for one.