- Arithmetic intensity
- The number of computations performed per number fetched from memory. High intensity means a workload is compute-bound and can approach the arithmetic ceiling; low intensity means it is memory-bound and the units starve. Batching raises intensity, which is why it is the standard remedy for low utilization in serving.
- Collective operation
- A coordinated exchange in which many chips combine their partial results before any can proceed. It acts as a synchronization barrier whose cost is set by the slowest path through the fabric, so every chip waits on it, making network topology and parallelism strategy first-order determinants of cluster throughput.
- Goodput
- The fraction of a cluster's time that produces useful, kept work, as opposed to time spent idle, recovering, or redoing work a failure destroyed. It can fall below instantaneous utilization because failures force restarts, and it is the more honest predictor of finished work over the long runs that scale demands.
- High-bandwidth memory
- Memory dies stacked and bonded directly beside the compute die on the same package, connected by a very wide bus. Its physical closeness enables the enormous bandwidth modern accelerators need; the advanced packaging that attaches it is difficult and supply-constrained, making it one of the scarcest components in the chain.
- Memory wall
- The widening gap between how fast an accelerator can compute and how fast it can move data in and out of memory. Because arithmetic has outgrown bandwidth for decades, the chip is chronically starved for data, making memory bandwidth, not arithmetic, the governing constraint on most workloads.
- Peak vs. sustained performance
- Peak is the theoretical maximum an accelerator can compute with every arithmetic unit busy and ideal data in place; sustained is the work actually delivered once stalls, data movement, and coordination are accounted for. The ratio between them, utilization, is usually well below the ceiling and is set by the surrounding system, not the die.
- Scale-up domain
- A small group of accelerators bound by dedicated high-bandwidth, low-latency links so they can read each other's memory almost as their own and behave like one large chip. Its size is a critical design parameter: work that stays inside runs fast, and work that crosses out drops to the slower scale-out tier.
- Utilization
- The fraction of an accelerator's peak that a real workload realizes. It is bounded by how well memory, interconnect, and software keep the arithmetic units fed, and it is the primary lever an operator can move through batching, scheduling, and data staging without buying more hardware.