The Bandwidth Wall: Why Memory, Not Math, Governs the Accelerator
Start with the correct mental model. A modern AI accelerator is not a compute engine that occasionally reads memory; it is a memory engine that occasionally computes. Peak floating-point throughput has grown roughly an order of magnitude faster than the bandwidth available to feed it, so most large-model kernels sit idle waiting for operands. My read: once you accept that the die is starved, every architectural decision downstream, from HBM stack height to how you batch inference, becomes legible. The bandwidth wall is not a temporary bottleneck. It is the governing constraint of the era, and GPU generations set its pace.
Knowledge check
What does it mean to call a large-model kernel 'memory-bound,' and why is that the common case on modern accelerators?
Show the answer
A kernel is memory-bound when its runtime is limited by how fast operands can be moved from memory to the compute units, not by arithmetic throughput. It is common because FLOPS have scaled far faster than memory bandwidth, so the arithmetic units finish quickly and then stall waiting for the next tile of weights or activations to arrive.