Inside the Accelerator
An accelerator is three systems on one package: arithmetic, memory and links. The arithmetic is quoted loudest and constrains least. Designers can add matrix units faster than they can feed them, which is the memory wall Wulf and McKee named in 1995, now arriving at the scale of a data center. High-bandwidth memory narrows the gap on one chip; scale-up links narrow it across many. Faraday's read is that memory capacity, memory bandwidth and the size of the scale-up domain explain more about delivered performance than any headline compute figure. Read the specification sheet from the bottom up.
Knowledge check
What is the memory wall, and how does high-bandwidth memory address it?
Show the answer
The memory wall, named by William Wulf and Sally McKee in 1995, is the ceiling created when processors improve faster than memory can supply them with data, so the processor spends its time waiting. High-bandwidth memory stacks memory dies beside the processor on the same package and connects them over a very wide path, raising the rate at which data reaches the arithmetic units. It narrows the gap but does not remove it, which is why large models are still split across many chips.