The Small Trigger and the Latent Weakness

If you read enough public outage reports, a suspicion hardens into a conviction: the size of the trigger tells you almost nothing about the size of the outage. Tiny triggers produce catastrophes; large-seeming events produce shrugs. The variable that actually governs severity is not the trigger at all. It is the state of the system the trigger happens to strike. This lesson is about the most important recurring pattern in the entire field, the one every other pattern is a special case of: a small trigger meeting a latent design weakness and being amplified out of all proportion.

Start with the amplification itself, because it is where the drama comes from. In a well-behaved system, a small perturbation produces a small, contained effect and then dies out. Push it a little, it wobbles, it settles. In a system carrying a latent weakness, the same small push does not settle; it grows. Some mechanism in the system takes the initial disturbance and feeds it back on itself, so the effect gets larger rather than smaller, and what began as a blip becomes a cascade. The feedback loop is the amplifier, and latent weaknesses are very often loops that nobody realized were there, or nobody realized were positive rather than negative.

Let me give you the shape in the abstract, because the abstraction is what transfers. A system is under load. Something causes a small number of operations to slow down or fail. In a forgiving design, those failures are absorbed. But suppose the system responds to a failed operation by retrying it, immediately and aggressively, as many systems sensibly do. Now imagine a moment when many operations fail at once. Each failure spawns retries, the retries add load, the added load causes more operations to fail, those failures spawn more retries, and the system has walked itself into a storm that it is now actively feeding. Notice that nothing is broken in the ordinary sense. Every component is doing exactly what it was designed to do. The retry logic that is a virtue under small stress becomes a vice under large stress, and the transition between the two is sudden. That suddenness, a system that is fine, fine, fine, and then abruptly not fine, is the signature of a latent weakness reaching its tipping point.

The reason this pattern is worth memorizing is that the same skeleton wears endless costumes. Sometimes the amplifier is retries. Sometimes it is a cache that, once it empties, sends a flood of traffic to a database that was only ever sized for the trickle that missed the cache, so the moment the cache fails the database is crushed, which keeps the cache from refilling, which keeps the database crushed. Sometimes it is a queue that backs up faster than it drains, so latency climbs, which causes timeouts, which causes resubmissions, which lengthens the queue. Sometimes it is a health-checking system that, under stress, marks healthy things as unhealthy and removes them, reducing capacity, increasing stress, and marking still more things unhealthy. Different mechanisms, one shape: a loop that was stabilizing under normal conditions becomes destabilizing under abnormal ones.

Now layer in the trigger, and appreciate how little it matters. The cache can empty because of a bug, a deployment, a hardware fault, or a random traffic spike; the ensuing catastrophe is identical regardless of which. This is why cataloguing triggers is a fool's errand and why I keep returning to the latent condition. The condition, an unprotected feedback loop with no circuit-breaker, is a fixed, findable property of the design. The trigger is a lottery. You cannot win the lottery reliably, but you can go audit your system for feedback loops that lack a governor, and that audit is entirely within your power. Faraday's read: when you see a report describing a sudden, disproportionate cascade, do not fixate on the initiating event. Find the amplifier, because the amplifier is the lesson and the amplifier is what your own system might be hiding.

There is a second, quieter reason these patterns recur across companies that never share code or staff. The latent weaknesses are not accidents of sloppiness; they are the natural sediment of growth and optimization. Retries make systems more reliable, so we add them. Caches make systems faster, so we add them. Health checks make systems self-healing, so we add them. Each is a genuine improvement under the conditions it was designed for, and each quietly introduces a feedback loop that becomes dangerous under conditions no one was thinking about yet. Complex systems drift toward these weaknesses precisely because each individual step was reasonable. That is the humbling part, and it is why blamelessness is not sentimental but accurate: nobody installed the weakness on purpose. It accumulated, one sensible optimization at a time, until a small trigger found it. Your job is to find it first.