- AI factory
- A framing of the modern AI data center as a production line whose single job is to convert electricity and data into finished tokens at the lowest cost and highest sustained rate. It imports manufacturing disciplines — throughput, line balancing, bottleneck analysis, and yield — into infrastructure design.
- Goodput
- Useful output actually delivered, as opposed to raw throughput, which counts all work including work later discarded to failures, stragglers, and reruns. It is the factory's notion of yield — good units shipped rather than units merely produced — and protecting it is why failure domains, checkpointing, and redundancy matter at cluster scale.
- High-bandwidth memory (HBM)
- Memory built from stacked dies mounted in the same package as the compute die and connected by very wide interfaces, so operands reach the arithmetic units far faster than ordinary board-mounted memory could deliver them. It exists specifically to help narrow the memory wall.
- Memory wall
- The gap between how fast an accelerator can perform arithmetic and how fast memory can supply operands. It leaves the arithmetic array starving between operations and is the reason high-bandwidth in-package memory and reduced numeric precision are central to accelerator design.
- Model FLOPs utilization (MFU)
- The ratio of the arithmetic a model genuinely needed to the arithmetic the hardware could have performed at full speed in the same time. Typical well-run values of roughly forty to fifty percent expose how much of a building's peak specification is never realized, and closing the gap is the aim of density, fabric, locality, and orchestration.
- Scale-up versus scale-out
- The two tiers of the fabric. Scale-up tightly couples a small group of accelerators into one very high-bandwidth domain so they behave almost like a single large chip; scale-out connects many such groups into a large cluster over data-center networking. Good architecture keeps the heaviest traffic inside the faster scale-up tier.
- Tokens-per-dollar
- The capital-and-operating efficiency ratio: realized token output divided by the fully loaded cost of producing it (silicon, building, power, cooling, network, staff, and financing). Raising utilization is its most powerful lever, since idle silicon depreciates whether or not it computes.
- Tokens-per-watt
- The energy-efficiency ratio: realized token output divided by the power consumed to produce it. As power becomes the scarce, capped input, this ratio becomes the master variable for how much a fixed megawatt allotment can produce, so every watt lost to heat movement or data movement is a token forgone.