Accelerators as Machine Tools: Arithmetic, Memory, and Precision

The machines on the AI factory floor are accelerators — GPUs, TPUs, and a growing field of custom ASICs — and understanding them starts with rejecting the idea that they are general-purpose computers. They are machine tools, purpose-built for one operation performed at enormous scale: dense linear algebra, the matrix multiplications that constitute nearly all of a large model's arithmetic. Everything about their design bends toward doing that one operation fast and feeding it without starving.

The heart of an accelerator is an array of arithmetic units engineered for multiply-accumulate — the operation at the core of a matrix multiply. Google's design uses a systolic array, a grid of units through which data flows rhythmically, each cell multiplying and passing partial sums to its neighbor so that operands are reused in place rather than repeatedly fetched. GPUs reach the same end differently, with thousands of small cores organized into tensor units that chew through matrix tiles. The specifics differ; the principle is identical. Pack as many multiply-accumulate units as the silicon and power budget allow, keep them busy, and reuse each fetched number as many times as possible before spending energy to move another one.

That last clause exposes the real constraint. Modern accelerators can perform arithmetic far faster than any memory can supply operands. The gap between how fast a chip can compute and how fast it can be fed is the memory wall, and it dominates accelerator design. The response is high-bandwidth memory, HBM — stacks of memory dies mounted in the same package as the compute die, connected by extraordinarily wide interfaces. HBM exists because ordinary memory, sitting across a circuit board, cannot deliver operands quickly enough to keep the arithmetic array fed. A leading accelerator pairs its compute with several stacks of HBM precisely so the machine tool does not starve. When people say an accelerator is "memory-bound," they mean the arithmetic array is finishing its work and waiting for the next operands — the machine tool is idling between parts because the feeder cannot keep up.

Two design levers follow directly. The first is the on-chip memory hierarchy: small, fast pools of memory close to the arithmetic units that hold operands being actively reused. Good model code stages data through these pools so a weight fetched once is used many times before it is discarded — the exact reuse-in-place logic of the systolic array, expressed in software. The second lever is numeric precision. Lower-precision formats — smaller representations of each number — halve or quarter the bytes moved and the energy per operation, and modern accelerators include dedicated hardware for them. Training and inference increasingly run in reduced precision wherever accuracy tolerates it, because precision is a direct dial on both the memory wall and the energy bill: fewer bits per number means more numbers delivered per unit of bandwidth and more arithmetic per watt.

Custom silicon is the natural endpoint of this specialization. A general GPU carries hardware for graphics and a broad range of workloads; an ASI built only for model arithmetic can spend every transistor and every watt on the operation that matters. Hyperscalers design their own accelerators for exactly this reason — to strip out the general-purpose overhead and tune the memory system, the numeric formats, and the interconnect to their own model shapes. The trade is flexibility for efficiency. A custom chip that fits one workload perfectly wins on tokens-per-watt; it loses if the workload shifts beneath it. This is a manufacturing decision in disguise: a bespoke machine tool built for one part is unbeatable on that part and useless on the next, so you build custom only where volume and stability justify sinking the design cost.

For the factory frame, the accelerator is one station on the line, and the lesson is that its usable output is set by feeding, not by its arithmetic peak. A machine tool rated for a thousand parts an hour delivers that only if the conveyor and the material handler keep pace; otherwise it idles, and you paid for capacity you cannot use. The accelerator's headline arithmetic rate is the thousand-parts rating. Whether the building realizes it depends on HBM bandwidth, on-chip reuse, precision choices, and — as the next lessons show — the density and cooling that let the chip run at full clock, and the fabric that connects it to its peers. The chip is necessary and never sufficient. It is the fast station whose value is unlocked, or wasted, by the rest of the line.