Why FLOPS Outran Bandwidth

The bandwidth wall exists because two quantities that need to grow together have grown at different rates for structural reasons rooted in physics and packaging, not oversight. Understanding why the gap opened tells you why it will not close on its own.

Compute throughput scales with transistor count and clock frequency, and for decades both improved with process shrinks. Adding arithmetic units to a die is comparatively cheap: they are small, they are regular, and shrinking the process gives you more of them in the same area. Vendors also learned to lower numerical precision, moving from thirty-two-bit to sixteen-bit to eight-bit and now four-bit formats for the multiply-accumulate operations at the heart of neural networks. Each halving of precision roughly doubles effective throughput for the same silicon. The result is that headline FLOPS numbers have climbed steeply, generation over generation, and the marketing has followed them.

Memory bandwidth obeys a harsher arithmetic. Bandwidth is the product of interface width, the number of parallel data wires, and the signaling rate per wire. Both are hard to scale. Signaling rate is limited by the physics of pushing clean electrical pulses across a channel; past a point, faster signaling demands disproportionate power and elaborate equalization to keep the eye open. Interface width is limited by pins and by the beachfront of the die edge, the finite perimeter available to route wires off the chip. You cannot simply add ten thousand more wires to a package. So bandwidth improvements have come in hard-won increments, and they have not kept pace with the near-exponential climb of compute.

This gap has a name in the older literature, the memory wall, and it long predates AI. What AI did was make the wall the dominant cost rather than an annoyance. Traditional workloads had enough locality, enough reuse of data already sitting in on-chip caches, that the slow path to main memory was hidden most of the time. Large neural networks broke that assumption. The weight matrices are enormous, they are read in full for each forward pass, and during single-stream inference they are barely reused before being discarded. There is nowhere for caches to hide the traffic. The workload puts the memory system directly on the critical path and leaves it there.

High-bandwidth memory is the industry's structural answer, and it is worth seeing it as a packaging innovation as much as a memory innovation. Instead of placing a few memory chips at the far end of a long circuit-board trace, HBM stacks memory dies vertically into a cube, connects the layers with vertical wires that run straight through the silicon, and places the whole stack a few millimeters from the processor on a shared substrate. That short distance and enormous wire count buys a very wide, very fast interface. It attacks the width side of the bandwidth equation by exploiting the third dimension, going up rather than out, because the die's two-dimensional edge had run out of room. We will dissect the mechanics in the next module.

But recognize the limit even as you admire the fix. HBM widens the road; it does not repeal the arithmetic. Compute still scales faster than bandwidth within each generation, because the underlying reasons, cheap regular arithmetic units against expensive constrained interfaces, have not changed. Each new HBM generation restores some ground, and each new compute generation gives it back by raising the ceiling above it. The ridge point ratchets rightward. My read: this is a permanent condition of the technology, not a phase, and the only durable strategy is to treat bandwidth as the scarce resource it is and to design models, kernels, and serving systems around minimizing bytes moved per unit of useful output.

The throughline for the course sits here. Because compute and bandwidth are co-designed in the same GPU generation, and because the vendor decides how much of each to ship, the balance you must optimize against is not something you choose. It is inherited. When a new generation lands with more compute and proportionally less bandwidth gain, every serving economics spreadsheet in the industry shifts on the same day, in the same direction, for the same reason. That is what it means to say the generation causes the downstream consequences. Hold that causal arrow. The remaining modules trace it through the physical memory, through the companies that manufacture it, and into the dollar cost of a generated token.