Two Networks: Scale-Up and Scale-Out

Inside a large AI cluster there is not one network but several, organized into a hierarchy, and the hierarchy exists because communication at different scales has different physics. The two most important tiers go by the names scale-up and scale-out, and understanding the difference is the key to understanding modern AI infrastructure.

Scale-up is the network inside a single server or a small group of tightly-coupled servers. It connects a handful of accelerators directly to each other with links of extraordinary bandwidth, so that this small group behaves almost as if it were one enormous chip. The defining example in the current market is the high-bandwidth interconnect that one dominant accelerator vendor uses to lash its chips together within a server, a technology often referred to by the brand name for its chip-to-chip links. The bandwidth here is far higher than anything available across longer distances, because the chips are physically close and the links are short, dense, and expensive. Scale-up is about making a small number of accelerators act as a single powerful unit.

Scale-out is the network that connects those units to each other, joining many servers into a cluster of hundreds or thousands of accelerators. Here the distances are longer, spanning racks and rows within a data-center hall, and the technology is switched networking. Two families dominate. One is InfiniBand, a networking standard long used in high-performance computing, prized for low latency and a design that assumes the network will rarely drop data. The other is Ethernet, the ubiquitous standard of general networking, now being adapted specifically for AI workloads by an industry effort producing a specification often described under the banner of Ultra Ethernet. The contest between these two families is one of the defining commercial battles in AI infrastructure.

Why split the network this way? Because bandwidth is cheapest and highest over short distances and grows more expensive and more constrained as distance increases. It would be wonderful to connect every accelerator in a cluster to every other with the bandwidth available inside a single server, but the physics and the cost forbid it. So the system is built as a hierarchy: lavish bandwidth among a few nearby chips, then progressively less as the scope widens. The art of cluster design is arranging the workload so that the heaviest communication happens where the bandwidth is richest, within the scale-up domain, and lighter communication spans the scale-out network.

This hierarchy shapes how models are split across hardware. When engineers decide how to divide a model, they are really deciding which communication happens over the fast short links and which happens over the slower long ones. Partitions that require constant, heavy exchange are kept within a scale-up group where bandwidth is abundant. Partitions that can tolerate less frequent exchange are spread across the scale-out fabric. The mapping between a model's structure and the network's hierarchy is one of the most consequential decisions in large-scale training, and it is invisible to anyone counting only chips.

The hierarchy also explains why the size of a scale-up domain matters so much commercially. The more accelerators a vendor can connect at scale-up bandwidth, the larger the unit that behaves like a single chip, and the easier it becomes to train large models efficiently. Expanding the scale-up domain is therefore a central axis of competition, and vendors invest heavily in the switching and the links that let more chips join a single high-bandwidth group. A larger scale-up domain is worth real money because it relaxes the hardest constraint in model partitioning.

The durable point is that an AI cluster is a layered communication system, that the layers reflect the economics of bandwidth over distance, and that the placement of a workload within those layers determines how efficiently the whole machine runs. Two clusters with identical chip counts can differ enormously in useful performance because their fabrics, and the way work is mapped onto them, differ. The fabric is not plumbing beneath the computer. The fabric is the computer's nervous system, and its architecture is as consequential as the chips it connects.