Escape Routes: CXL Memory Pooling and Processing-in-Memory

If bandwidth and memory capacity near the compute govern inference cost, then two ideas attack the problem from opposite directions. One expands the pool of memory a processor can reach; the other moves computation into the memory so the data need not travel. Neither repeals the bandwidth wall, but each changes where it binds, and both are drawing serious investment for that reason.

The first is Compute Express Link, a fast interconnect standard that lets a processor attach and share memory beyond what sits directly on its package. CXL runs over the same physical lanes used for high-speed peripheral connections and adds coherency, so attached memory can be addressed much like local memory. Its promise is capacity and flexibility: a system can attach large pools of memory over CXL, and those pools can be shared among several processors rather than stranded on one. In a world where the intermediate-state cache and large models strain on-package capacity, an expandable tier of memory relieves pressure, letting you hold more without buying more accelerators purely for their attached memory.

But be precise about what CXL does and does not fix. Memory reached over CXL is farther away than on-package HBM, so it offers more capacity at lower bandwidth and higher latency. It is a capacity tier, not a bandwidth tier. It helps when the problem is fitting something large into memory at all, or sharing a memory pool flexibly across machines; it does not help the memory-bound decode kernel that needs weights streamed at HBM speed. The right mental model is a hierarchy: fast, scarce HBM close to the die for the bandwidth-critical traffic, and a larger, slower CXL-attached pool behind it for capacity and pooling. CXL widens the memory system's back tiers; it does not raise the ceiling on the roofline. My read: it is a valuable pressure-relief valve and a poor substitute for on-package bandwidth, and treating it as either extreme will mislead you.

The second idea is more radical: processing-in-memory, which places simple computation inside or immediately beside the memory itself. The motivation is that moving data is more expensive, in time and energy, than computing on it, so if some operations can be performed where the data already lives, the round trip to the processor is avoided entirely. For memory-bound operations with low arithmetic intensity, exactly the operations that define decode, doing the work in the memory can sidestep the bandwidth bottleneck rather than merely widening it. The memory makers have built experimental parts that embed arithmetic units among the memory banks, targeting precisely the low-intensity, bandwidth-hungry steps that starve conventional accelerators.

The obstacles are real and worth naming. Adding logic to memory dies is hard because memory and logic are fabricated with different, specialized processes, and combining them compromises one or both. The programming model is unsettled: software and compilers assume computation happens in the processor, and moving some of it into memory requires new abstractions the ecosystem has not standardized. And processing-in-memory helps only a specific class of operations, so it is a targeted tool rather than a general accelerator. It is promising precisely for the memory-bound heart of inference, and unproven at scale, and both things are true at once. My read: watch it as a genuine structural response to the bandwidth wall rather than a near-term product, and note that the base logic die trend in HBM4 is a quiet step in the same direction, putting more capability at the memory's edge.

Step back and hold the whole chain in view, because it is the point of the course. A GPU generation is designed with a compute ceiling and, beside it, an HBM generation setting a bandwidth ceiling. That memory is manufactured by three firms under yield and capacity constraints that make it scarce and its allocation strategic. The resulting bandwidth and memory cost flow into the memory-bound economics of decode, where they set the price of a generated token. And the escape routes, pooling memory over a fabric, moving compute into memory, are attempts to bend that chain without breaking the physics beneath it. Every link is caused by the one above it, and the top link is the generation. That is what it means to say GPU generations cause every downstream consequence: the phrase is not rhetoric but a literal description of the causal structure you have now traced end to end, from arithmetic intensity to the dollar cost of a word.