Bandwidth and Latency: The Two Numbers That Govern Everything

Two quantities govern network behavior, and confusing them is the most common error in the field. The first is bandwidth: how much data a link can move per unit of time. The second is latency: how long a single piece of data takes to travel from one end to the other. They are independent, and a system can be rich in one and starved of the other.

An analogy fixes the distinction. Bandwidth is the width of a pipe; latency is the length of the pipe. A very wide pipe can carry an enormous volume of water per second, but if it is also very long, any particular drop still takes a while to arrive. Widening the pipe does nothing to shorten the trip. This is why you cannot buy your way out of a latency problem with more bandwidth, and why the two must be reasoned about separately.

Bandwidth is set by the physics of the link and the electronics at each end. A modern optical link carries light modulated at enormous rates, and the transceivers that generate and read that light are among the most demanding components in the whole system. When people speak of network capacity in a data center, they are usually speaking about bandwidth: the aggregate volume the fabric can shift.

Latency has two sources. The first is unavoidable and set by nature: light and electrical signals travel at a finite speed, so distance imposes a floor. Signals in fiber move at roughly two-thirds the speed of light in vacuum, which works out to a delay of about five microseconds for every kilometer of fiber. Across a data-center hall this is negligible. Across an ocean it is not. A signal crossing between continents accrues tens of milliseconds of delay that no engineering can remove, because the distance is fixed and the speed of light is a constant.

The second source of latency is added by the system itself: every switch a signal passes through, every conversion between optical and electrical form, every moment spent waiting in a queue behind other traffic. This portion is where engineering lives. Network designers work relentlessly to reduce switch hops, shorten queues, and keep the fabric uncongested, because this is the latency they can actually control.

Why do these two numbers govern what a cluster can do? Because different parts of an AI workload are sensitive to different ones. When accelerators synchronize after a training step, they perform a collective operation in which every chip must both send and receive before any can proceed. The slowest link in that operation sets the pace for all of them. If the collective is dominated by large data transfers, bandwidth rules. If it is dominated by many small coordinating messages, latency rules, because each message pays the full travel-time toll and the tolls add up.

This is why a cluster's effective performance cannot be read off a single specification. A fabric with abundant bandwidth but poor latency will handle bulk transfers well and choke on tightly-coupled coordination. A fabric with low latency but limited bandwidth will coordinate crisply and then stall when it has to move large volumes. Real workloads mix both patterns, and the network has to satisfy both at once.

There is a further subtlety worth naming: consistency matters as much as peak. A link that is usually fast but occasionally slow, a phenomenon network engineers call tail latency, can wreck a collective operation, because the operation waits for the slowest participant every single time. A fabric is only as good as its worst-behaving link under load. This is one reason the back-end networks that connect accelerators are engineered for predictability, not merely for headline speed. Predictable mediocrity often beats unpredictable excellence when thousands of chips must march in lockstep.

Hold these two numbers in mind for the rest of the course. Every layer we examine, from subsea cable to in-rack link, is ultimately an attempt to deliver enough bandwidth at low enough and consistent enough latency to keep expensive silicon busy. When a network layer fails to deliver AI compute, it is almost always because one of these two numbers fell short, or because the two could not be satisfied together.