Memory Systems: HBM and the Bandwidth Wall
Glossary
- Arithmetic intensity
- The ratio of floating-point operations performed to bytes moved from memory for a given kernel. Compared against a chip's compute-to-bandwidth ratio, it predicts whether the kernel is limited by math or by memory.
- Compute Express Link (CXL)
- A coherent high-speed interconnect standard that lets a processor attach and pool memory beyond its package. It provides a larger, slower capacity tier behind on-package HBM, not additional bandwidth for memory-bound kernels.
- HBM4
- The next HBM generation, defined by doubling the interface width to roughly 2048 data lines per stack and by a more capable base logic die, raising bandwidth without pushing per-pin signaling rates into a power-prohibitive regime.
- High-bandwidth memory (HBM)
- Memory built by stacking multiple dies vertically, connecting them with through-silicon vias, and placing the stack beside the processor on a silicon interposer to expose a very wide, fast interface at low power per bit.
- Memory-bound kernel
- A kernel whose runtime is limited by how fast operands can be delivered from memory rather than by arithmetic throughput. LLM decode, which streams full weights per token, is the canonical example.
- Roofline model
- A performance diagram with a sloped memory-bandwidth ceiling and a flat compute ceiling. The ridge point where they meet is the arithmetic intensity a kernel must reach to stop being memory-bound.
- Silicon interposer
- A slab of silicon beneath both memory stacks and processor die carrying a dense mesh of fine lithographic wiring, enabling a thousand or more data lines to run between them across a few millimeters. The basis of 2.5D integration.
- Through-silicon via (TSV)
- A metal-filled vertical column etched straight through a thinned memory die, carrying signals and power between stacked layers along the shortest path instead of routing around the die edge.