Two Tiers, Two Physics

The fabric is not one network. It is two, stacked, obeying different physics and solving different problems. Confuse them and nothing about cluster design makes sense. Separate them and the whole architecture falls into order.

The first tier is scale-up: the network inside a tightly coupled group of accelerators — historically a single server, increasingly a whole rack engineered as one unit. Here the goal is to make many processors behave as if they were one very large processor. The accelerators in a scale-up domain are joined by the fastest, shortest, most expensive links available, so tightly that one can reach into another's memory almost as if it were local. This is the domain of NVLink and NVSwitch, the subject of the next module. Its defining property is bandwidth so high and distance so short that the group approximates a single coherent machine.

The second tier is scale-out: the network that joins those groups into a cluster of thousands or tens of thousands. Here the links are longer, the participants far more numerous, and the physics harsher — you are crossing a room, then a building. This is the domain of InfiniBand and Ethernet, the subject of Module 3. Its defining property is reach: it connects everything, but at lower bandwidth and higher latency per link than the scale-up tier.

Why two tiers at all? Because bandwidth is cheap over centimeters and ruinously expensive over tens of meters. A signal degrades with distance; driving it far enough, fast enough, costs power, silicon, and — past a point reached quickly at these speeds — a conversion from electricity into light and back. You cannot afford to wire ten thousand accelerators to each other the way you wire eight. So you build a small, dense, blindingly fast domain where you can, and a larger, slower, sparser domain to connect those domains where you must. The two-tier structure is not a preference. It is what the economics of moving a bit over a distance forces on you.

This produces a hierarchy of communication cost that governs how models are mapped onto hardware. Talking to an accelerator in your own scale-up domain is fast and abundant. Talking to one across the scale-out fabric is slower and scarcer. The consequence is a design rule: keep the most frequent, most bandwidth-hungry communication inside the scale-up domain, and let only the less frequent traffic cross the scale-out tier. The way a model is partitioned across the machines is, in large part, an exercise in respecting this hierarchy — placing the chattiest relationships where the wire is fattest.

This is why the size of the scale-up domain is one of the most consequential numbers in the whole system, and one of the least discussed outside the engineering rooms. The more accelerators you can bind into a single high-bandwidth domain, the more of your expensive communication stays on the fast tier, and the less is exiled to the slow one. Widening the scale-up domain — going from a handful of accelerators to a rack-scale group acting as one — moves the boundary between fast and slow communication outward, and moves utilization up with it. Faraday's read: the race that decides cluster performance is not only the accelerator race. It is the race to widen the coherent domain — to make the single fast machine at the center of the cluster as large as physics and money allow.

Notice what this does to the mental model most people carry. The instinct is to picture a cluster as a flat field of identical processors, each equivalent to the next. The reality is a landscape with sharp elevation: tight, fast valleys of scale-up connectivity, joined by longer, slower roads of scale-out fabric. Two accelerators in the same valley are near neighbors; two in different valleys are distant, whatever the floor plan says. Distance in a cluster is measured in bandwidth and latency, not meters. The map that matters is the network topology, and it is not flat.

So the fabric decomposes into two questions, and the next two modules take them in turn. How large and how fast can you make the coherent scale-up domain — the single machine at the core? That is NVLink and NVSwitch. And how efficiently can you join those domains into a cluster without the tail exploding — how do you move ten thousand into rough agreement? That is InfiniBand, Ethernet, and the switch silicon underneath both. Two tiers, two physics, one machine. Keep them distinct, and the rest of the course is a map you can already read.