Scarcity, Yield, and the Economics of Sold-Out Memory
High-bandwidth memory is scarce for reasons that are structural and slow to change, and that scarcity propagates outward until it becomes a governor on the entire AI buildout. Trace the scarcity to its sources and you see why it will persist.
The first source is yield. Stacking eight, twelve, or sixteen thin dies into a single part and wiring them with thousands of vertical connections is unforgiving. A defect in any layer, or a bad connection anywhere in the stack, can ruin the whole assembly, so the fraction of good parts coming off the line is materially lower than for ordinary flat memory. Because every good die that goes into a stack could otherwise have been sold on its own, HBM also consumes far more wafer capacity per usable gigabyte than commodity memory. The manufacturing is slow, the good-part rate is hard-won, and improving it is a matter of painstaking process learning rather than a switch anyone can flip.
The second source is that HBM competes with ordinary memory for the same fabrication capacity. Every wafer devoted to HBM is a wafer not making standard memory, so a surge in HBM demand tightens the broader memory market too. The makers must decide how much of their finite advanced capacity to commit to HBM versus everything else, and because HBM carries premium margins during the AI surge, they have tilted hard toward it, which ripples through memory pricing generally. This coupling is why the AI buildout has affected the price of memory in devices that have nothing to do with AI.
The third source is time. New capacity for advanced memory takes years to build and qualify. You cannot respond to a demand spike by opening a fab next quarter; the equipment lead times, cleanroom construction, and process ramp run on multi-year clocks. So when demand jumps, as it did when large-model training and inference scaled, supply cannot chase it quickly, and the gap is closed by allocation and by price rather than by volume in the near term. This is the classic profile of a market that sells out and stays sold out for an extended stretch.
There is a fourth source that compounds the others: HBM demand per accelerator keeps rising. Each GPU generation tends to want more memory and taller stacks than the last, so even if the number of accelerators shipped held flat, the HBM required to build them would climb. Demand is therefore growing on two axes at once, more accelerators and more memory per accelerator, while supply grows on the single slow axis of fab construction. That structural mismatch is why the shortage has not been a brief spike but a sustained condition, and why makers have been willing to commit enormous capital to new capacity despite the memory industry's long history of ruinous boom-and-bust cycles. They are betting that this demand is durable rather than cyclical, and the length of the current advance-purchase queues is the evidence they are pointing to.
The consequence is that leading-edge HBM is contracted far in advance, often a year or more, with the large accelerator buyers locking up output before it exists. In that regime the binding question is not price per gigabyte but whether you can get an allocation at all. An accelerator maker that has secured memory can ship; one that has not, cannot, regardless of how good its silicon is. Memory availability becomes a gate on the whole product. My read: this is why memory supply, not chip design, is frequently the true limiter on how many accelerators reach the market in a given year. The compute die may be ready, the packaging capacity may be ready, but if the qualified HBM is not there in volume, the systems do not ship.
Carry the logic to its end and the course thesis reappears in economic form. The GPU generation sets how much bandwidth each accelerator wants, which sets how much HBM each unit consumes, which, against a supply that cannot flex quickly, sets how many units can exist and at what cost. A generation that demands more or taller stacks per accelerator tightens the memory market further and pushes the cost of every unit up, and that cost flows straight into the price of serving a model. The scarcity is not a temporary supply-chain hiccup to be waited out; it is the steady-state condition of a market where demand scales with the AI roadmap and supply scales with fab construction timelines, and the two run at different speeds. Understanding this lets you predict the direction of accelerator availability and cost from the memory news alone, because in this industry the memory news arrives first and the consequences follow. The final module turns that chain into dollars per token and into the architectures being built to escape it.