Common-Mode Failure and the Illusion of Redundancy

There is a particular kind of outage that produces a specific facial expression on the engineers involved: not panic, but a stunned disbelief, because the very thing that failed was the thing that was supposed to make failure impossible. This is the domain of common-mode failure, and it is the most instructive pattern in the field precisely because it defeats our primary defense against failure, which is redundancy. Understanding it will change how you look at every 'highly available' system you ever build or depend on.

Start with the promise of redundancy, because it is a good promise and mostly kept. If one component fails with some small probability, and you run two independent copies, then both fail together only with the product of their probabilities, a much smaller number. Run three copies and the number shrinks again. This is the arithmetic that underpins nearly all reliability engineering, and it is sound. But read the load-bearing word carefully: independent. The entire arithmetic rests on the assumption that the copies fail independently, that whatever takes down one has no bearing on the others. Common-mode failure is what happens when that assumption is false, when the copies share a hidden common cause that can take them all down at once. And when it is false, the reassuring arithmetic does not merely weaken. It collapses entirely, because a single common cause converts your three independent copies into one correlated thing wearing three hats.

The cruelty of common-mode failure is that the shared cause is usually invisible during normal operation, which is exactly when you would want to notice it. Consider what your redundant copies quietly have in common. They very likely run the same software version, so a bug in that version is present in all of them simultaneously; deploy the bad version everywhere and all your redundancy fails on the same instruction. They very likely read the same configuration, so a bad configuration pushed to all of them disables all of them at once, which is why configuration changes appear in so many outage reports as the trigger. They may depend on the same underlying service for something as mundane as authentication or name resolution or time, so when that shared dependency fails, every 'independent' copy fails together, because they were never independent below that layer. They may sit in the same building drawing the same power or cooled by the same system. Each of these is a common mode, a single fault line running underneath copies that looked separate from above.

Let me draw out the dependency version, because it is the most modern and the most treacherous. In a large system, services depend on other services, which depend on still others, and somewhere near the bottom of that pile sit a few foundational services that almost everything relies on, often without the people above realizing it. A name-resolution service, an identity service, an internal configuration store. These low-level dependencies are shared by design; that is what makes them efficient. But it also makes them common modes of staggering scope. When one of them fails, the failure does not stay contained; it radiates upward into every service that depended on it, and because nearly everything depended on it, nearly everything fails, seemingly at once, seemingly for unrelated reasons. Engineers on the bridge call see a dozen services failing and struggle to believe they share a cause, because from the top the services look independent. They were only ever independent above the waterline. Below it, they all stood on the same drowned foundation.

Here is the pattern that makes common-mode failure so pernicious, and I want you to hold it as a warning label on the word 'redundant': adding more copies does nothing against a common mode. If all your copies share the flaw, running more of them just means more things fail at the same instant. Redundancy multiplied against a common cause is redundancy wasted. This is why sophisticated reliability work is not really about adding copies; it is about hunting for correlation, about asking of any redundant set the deflating question, 'what do these supposedly independent things secretly share, and what happens when that shared thing fails?' The same version. The same config. The same dependency. The same power. The same deployment pipeline that can push a bad change to all of them in one motion. Every one of those is a common mode, and finding them is the actual work.

Faraday's read on why this recurs: common-mode failure is the shadow cast by every efficiency we love. Sharing a dependency is efficient. Standardizing on one software version is efficient. Deploying everywhere from one pipeline is efficient. Each act of sharing and standardizing improves the system on a normal day and installs a common mode for the abnormal one. So the pattern is not a sign of incompetence; it is the price of coherence, and it appears in report after report because every organization that grows large enough builds shared foundations, and shared foundations are shared fate. When you read one of these reports, resist the urge to feel clever about the victim's oversight. Instead, go find your own shared foundations, the services everything quietly stands on, and ask what your world looks like on the day one of them falls. That question, asked in advance, is the whole return on reading someone else's bad day.