Feeding the Chip: Memory and Interconnect
The most important number on an accelerator is not its arithmetic rate — it is how fast it moves data. Arithmetic has outgrown memory bandwidth for decades, so the chip is chronically hungry, and the high-bandwidth memory bonded beside the die, plus the advanced packaging that attaches it, is the scarcest link in the chain. The moment a model outgrows one chip, a second constraint takes over: the interconnect. A tightly bound group of accelerators behaves like one large machine; the same chips on ordinary links behave like a slow committee. So the real unit of design is the bound group, not the die — and both memory and interconnect cap the silicon from outside it.
Knowledge check
Why does serving a large model to one user at a time typically realize a low fraction of an accelerator's peak, and what is the standard fix?
Show the answer
Because single-user serving has low arithmetic intensity: you stream the model's entire set of parameters through the chip to produce only a few outputs, doing little computation per byte fetched, so memory bandwidth — not arithmetic — sets the pace and the units starve. The standard fix is batching many requests together, so each loaded parameter serves many computations at once, raising intensity and utilization.