Inside the Accelerator: The Arithmetic Engine

Model the accelerator as three systems sharing one package. Compute does the arithmetic. Memory holds the numbers. Links move numbers to other chips. A chip is as fast as the weakest of the three. This lesson covers compute. The next covers the other two.

Start with the building block. An NVIDIA graphics processing unit, or GPU, is organised into repeated units that NVIDIA calls streaming multiprocessors. Each one contains general arithmetic units and, since the Volta generation of 2017, dedicated matrix units that NVIDIA calls Tensor Cores. A matrix unit performs a small matrix multiplication as a single operation rather than as many separate multiplications and additions. Stack enough of them and you have a machine built around the one operation neural networks need most. The Instinct accelerators from Advanced Micro Devices, or AMD, and Google's tensor processing units, or TPUs, reach the same destination by different routes. Google's TPUs are built around large arrays of multipliers through which data flows in a fixed rhythm. The vocabulary differs. The principle does not: dedicate silicon to matrix math.

Next, the size limit. Chips are printed on silicon wafers by photolithography, and a lithography machine can expose only a limited area in one pass. The industry calls that limit the reticle. A single die cannot be larger than the reticle. For years, bigger chips meant getting closer to that ceiling. The ceiling has now been reached, so the industry has moved sideways. NVIDIA describes its Blackwell GPU, announced at its annual developer conference in March 2024, as two reticle-limited dies joined by a chip-to-chip link and operating as one GPU, with 208 billion transistors in total, manufactured by Taiwan Semiconductor Manufacturing Company, or TSMC. When one die cannot grow, you join two. The trade is that the join itself becomes a design problem.

Then, the packaging. An accelerator today is not a bare die. It is an assembly: compute dies, memory stacks and a base layer that wires them together at very short distances. That joining step is a bottleneck of its own. The best-known version is TSMC's chip on wafer on substrate process, or CoWoS, and on its 2025 earnings calls TSMC told investors it was still expanding CoWoS capacity to close the gap between demand and supply. For a non-engineer, one consequence is worth holding. Accelerator supply is limited by packaging and memory as well as by the logic chips themselves. A shortage can sit in any of the three.

Now the numbers that appear on every announcement. Performance is quoted in floating-point operations per second, abbreviated FLOPS, usually with a large prefix: teraflops for trillions, petaflops for thousands of trillions, exaflops for millions of trillions. Treat every such figure with three questions.

The first question is which number format. The same chip produces very different figures at 16-bit, 8-bit and 4-bit precision, because smaller numbers mean more operations per cycle. A headline figure at 4-bit precision is not comparable with an older chip's figure at 16-bit precision. NVIDIA's own page for its GB200 NVL72 rack, for example, quotes its headline compute at 4-bit precision. That is a legitimate figure. It is not a like-for-like comparison with a chip quoted at a higher precision.

The second question is peak or delivered. A peak figure assumes every arithmetic unit is busy every cycle. Real workloads rarely achieve that, because the units wait for data. The gap between the two is the subject of the next lesson.

The third question is dense or sparse. Some figures assume the model contains many zeros that the hardware can skip. That can be real for some models and irrelevant for others. Vendor footnotes usually say which applies. Read them.

For delivered performance, the best public reference is MLPerf, a set of benchmarks run by MLCommons, an industry consortium. Vendors submit results on defined training and inference tasks under published rules. NVIDIA, AMD and Google have all submitted results. MLPerf is not a verdict on which chip to buy. It is a common test that forces like-for-like conditions, and that alone makes it more useful than a peak number.

The compute side of the chip, then, is the part that improves most visibly and is quoted most loudly. Each generation adds matrix units, supports smaller number formats and packs more silicon into a package. Faraday's read is that compute is the least likely of the three systems to be the real constraint on a given workload. Designers can add arithmetic faster than they can feed it. That asymmetry has a name, and it is thirty years old.