Stacks, Through-Silicon Vias, and the Interposer

Take a high-bandwidth memory part apart conceptually and you find three ideas doing the work: vertical stacking, through-silicon vias, and a silicon interposer. None is exotic on its own. Together they solve the beachfront problem that ordinary memory packaging cannot.

Start with the stack. A single memory die holds a fixed number of storage cells, and its edge, the beachfront, has room for only so many connections. To get more capacity and more connections without a larger footprint, HBM stacks several dies on top of one another, typically eight, twelve, or now sixteen high in the leading parts. The stack sits on a base logic die that handles the interface to the outside world. Stacking multiplies capacity within the same tiny footprint and, crucially, multiplies the number of channels the stack can expose, because each layer contributes connections.

The layers are joined by through-silicon vias, vertical conductive columns that pass straight through the body of each die rather than routing signals around the edge. A via is a hole etched through the thinned silicon and filled with metal, so a signal can travel from the top die to the base die by going down through the stack, the shortest possible path. Thousands of these vertical connections run through a single stack. They are what make the third dimension usable: without a way to carry signal and power vertically through the silicon, stacking dies would just leave you with layers you could not efficiently talk to.

The stack still has to reach the processor, and here the silicon interposer does the work. An interposer is a slab of silicon that sits beneath both the memory stacks and the processor die, carrying an extremely dense mesh of fine wiring between them. Because the wiring is patterned in silicon with the same lithography used for chips, it can be far finer and far denser than anything on an ordinary circuit board. This density is what allows a thousand or more data lines to run from each memory stack to the processor across a gap of only a few millimeters. The processor and its memory become, in effect, a single package. This assembly style is often called 2.5D integration, memory and logic side by side on a shared silicon substrate, as distinct from true 3D stacking of logic on logic.

Now the payoff in numbers. Because the interface is enormously wide, on the order of a thousand data lines per stack in current generations, HBM achieves its bandwidth without demanding heroic per-wire speeds. A wide, moderately fast interface beats a narrow, frantically fast one on power, and power is the binding constraint in a dense accelerator. A single HBM3E stack delivers well over a terabyte per second of bandwidth, and an accelerator carries several such stacks around its perimeter, summing to multiple terabytes per second of aggregate bandwidth feeding the die.

Capacity rides along with bandwidth because both scale with stack height. Taller stacks, twelve-high and sixteen-high, hold more gigabytes and expose the memory needed to keep very large models resident near the compute. This matters for the workload we care about: the more of a model's weights you can hold in fast memory close to the die, the less you must page across slower links, and the more of your bandwidth budget goes to useful token generation rather than to shuffling data into place.

Thermal management is the quiet constraint threaded through all of this. Stacking dies traps heat: the layers in the middle of the cube are the hardest to cool because they are surrounded by other silicon, and memory grows unreliable as it heats. A taller, faster stack generates more heat in the same footprint, so packaging engineers must move that heat out through thinned dies and dense connections without room to spare. This is why bandwidth cannot simply be dialed up at will even when the interface would allow it; the stack must stay within a temperature envelope, and every gain in speed or height has to be paid for in the cooling budget. When you read that a new accelerator runs its memory at a given speed, that number is as much a thermal verdict as an electrical one.

My read: the reason to learn this anatomy is that every knob the GPU vendor turns to raise the bandwidth ceiling lives here. Add stack layers and you gain capacity and channels. Widen the interface and you gain bandwidth at fixed signaling rate. Improve the interposer and base die and you gain the ability to run all of it at speed within a power budget. When a vendor announces a new accelerator with more memory bandwidth, they are announcing decisions made at exactly this level of the stack. The bandwidth ceiling on your roofline is the sum of these physical choices, and it is handed to you fully formed with the GPU generation. You optimize against it; you do not set it. That is the causal arrow again, and it originates inside this cube.