The Memory Wall: Why Bandwidth Is the Real Currency
The single most important number about a modern accelerator is not how fast it computes. It is how fast it can read and write memory. This is the memory wall, and understanding it collapses most of the confusion about why chips underperform their datasheets. Arithmetic throughput has grown for decades at a far faster rate than the speed of moving data in and out of memory. The result is a chip that can compute far more than it can feed itself, so it spends its life hungry.
Picture the arithmetic array as a mouth and memory bandwidth as the size of the spoon. A bigger mouth — more arithmetic units — is useless if the spoon does not grow with it. On modern accelerators the mouth has outgrown the spoon by a wide and widening margin. This is why the memory stacked directly on the accelerator package, high-bandwidth memory, is the scarcest and most fought-over component in the entire supply chain. It is not the arithmetic that constrains the machine; it is the bandwidth of the memory bonded beside it.
High-bandwidth memory works by stacking memory dies vertically and bonding them next to the compute die on the same package, connected by an extremely wide bus. Width is the trick: instead of a narrow channel run at a blistering clock, you run a very wide channel at a moderate clock, moving enormous quantities of data per cycle. The physics of keeping that memory physically close to the compute — on the same package, within millimeters — is what makes the bandwidth possible, and it is also what makes it hard to manufacture, expensive, and supply-constrained. The advanced packaging that bonds these stacks together is itself a bottleneck upstream of the chip.
Two quantities govern how a workload interacts with this wall. The first is bandwidth: how many numbers per second can move between memory and the arithmetic units. The second is capacity: how many numbers the on-package memory can hold at once. They constrain different things. Capacity determines whether a model even fits — a model whose parameters exceed the memory on one accelerator must be split across several, which drags in the interconnect problem of the next lesson. Bandwidth determines how fast you can stream those parameters through the arithmetic units once they fit. A workload can be limited by either, and the discipline is knowing which.
The concept that ties it together is arithmetic intensity: the number of computations performed per number fetched from memory. If a workload does many operations on each value it loads, it is compute-bound and can approach the arithmetic ceiling. If it does few operations per value loaded, it is memory-bound and the arithmetic units starve no matter how many you have. Serving a large model to one user at a time is brutally memory-bound — you stream the entire set of parameters through the chip to produce a handful of outputs, doing very little arithmetic per parameter fetched. This is precisely why inference so often realizes a low fraction of peak: the workload's intensity is low, so bandwidth, not arithmetic, sets the pace.
The operator's response to a memory-bound workload is not to buy faster arithmetic — it does nothing — but to raise arithmetic intensity. Batching many user requests together so each loaded parameter serves many computations at once is the classic move; it converts a memory-bound job into a more compute-bound one and lifts utilization sharply. Keeping data resident on-package to avoid re-fetching, and choosing numerical formats that pack more values into the same bandwidth, are variations on the same theme: get more work out of each byte moved.
For the investor, the memory wall reframes the supply chain. The constraint on building capable systems is frequently not the compute die at all but the high-bandwidth memory and the advanced packaging that attaches it — a narrower, more concentrated supply base than the compute foundries. A company's ability to secure memory and packaging is as decisive as its access to the compute die. For the policy reader, the same logic sharpens the strategic picture: memory bandwidth and advanced packaging are chokepoints as real as the die itself, and a strategy that controls one while ignoring the others leaves a path to a working system open. The wall is the system's true edge, and everyone who builds at scale is pressed against it.