The Memory Hierarchy: Data Locality and the Real Bottleneck

The recurring enemy of the AI factory is not arithmetic; it is data movement. Doing a multiply is cheap. Getting the two numbers to the arithmetic unit at the right instant — from across the building, across the board, or across the chip — is expensive in both time and energy, and it is the true bottleneck of nearly every large-model workload. To reason about a building's throughput you have to reason about the memory hierarchy: the tiered system of storage that ranges from vast-but-slow to tiny-but-instant, and the cost of moving a number between the tiers.

Picture the hierarchy as a set of concentric rings around the arithmetic units. Innermost sits the on-chip memory — small pools of registers and fast local storage, close enough that operands arrive almost instantly and cost almost nothing in energy to reach. One ring out is the in-package high-bandwidth memory, larger but slower and more energy-costly to access. Further out is the memory of other accelerators in the same tightly-coupled group, reachable over the fast scale-up links. Furthest out is the memory of chips in other racks, reachable only across the slower scale-out fabric, and beyond that the storage systems holding datasets and checkpoints. The defining fact of this hierarchy is that each ring outward is dramatically slower and more energy-expensive to reach than the one inside it — often by an order of magnitude at each step. Moving a number from a neighboring rack can cost far more energy than the arithmetic that will consume it.

That asymmetry rewrites the optimization problem. If arithmetic were the cost, you would maximize arithmetic units. Because data movement is the cost, you instead minimize how far and how often data travels, and you maximize reuse of every number you have already paid to move. This is data locality, and it is the single most important idea in getting real output from an AI factory. Keep the operands a computation needs in the innermost ring you can, use each fetched number as many times as possible before evicting it, and structure the work so that chips talk mostly to their nearest neighbors rather than across the building. Every one of these moves attacks the same waste: the energy and time spent shuttling numbers instead of computing with them.

The techniques that dominate large-model engineering are, at bottom, data-locality techniques, though they wear other names. Tiling breaks a huge matrix multiply into small blocks sized to fit in fast on-chip memory, so each block's operands are fetched once and reused fully before the next block is loaded — the software expression of the reuse-in-place logic built into the hardware array. Operator fusion chains several arithmetic steps together so intermediate results never leave fast memory to be written out and read back. Careful placement maps the pieces of a model onto chips so that the pairs that must communicate most sit closest in the fabric hierarchy. Checkpoint and data-loading strategies stage the right bytes into the right ring ahead of when they are needed, so the arithmetic never waits on a distant fetch. None of these change the arithmetic; they change the data movement around it, and that is where the throughput and the watts are won or lost.

The energy dimension deserves its own emphasis because it is where tokens-per-watt is decided. In a modern accelerator, the energy to fetch an operand from a distant memory tier can exceed the energy to perform the operation that uses it by a wide margin. This inverts the naive intuition that computation is the expensive part. It means that in a power-capped building — which is increasingly every building — reducing data movement is not a refinement but the primary way to produce more tokens from the same watts. Two clusters with identical silicon and identical fabric can differ sharply in tokens-per-watt purely on how well their software keeps data local. Architecture here reaches into the software layer: the physical hierarchy sets the costs, and the code decides how often you pay them.

For the operator, the memory hierarchy is a lens on where the building actually spends its time and energy: profile for stalls waiting on data, and treat locality improvements as capacity increases that cost no new hardware. For the investor, a team's mastery of data movement — not just its chip count — separates buildings that approach their silicon's potential from those that squander it. For the policymaker, the fact that data movement often dominates energy use means efficiency gains in software and architecture, not only in chips, determine how much computation a fixed power supply can yield. The whole factory, from the arithmetic array outward through the fabric, is organized around one struggle: keep the machine tools fed while paying the smallest possible price to move the material. Win that struggle and the tokens follow.