Skip to the lesson

Major Outage Postmortems & Pattern Recognition

Glossary

Blameless postmortem
An incident review that deliberately refuses to stop at 'a person made a mistake,' treating human action as a normal event and asking instead what system, tooling, or organizational conditions allowed that action to cause harm. Its purpose is both accuracy and candor: people tell the truth about failures only when telling it is safe.
Common-mode failure
The simultaneous failure of components that were assumed to fail independently, caused by a hidden shared factor such as the same software version, configuration, dependency, deployment pipeline, or power source. It defeats redundancy, because adding more copies that share the flaw simply means more things fail at once.
Feedback loop (amplifying)
A mechanism in which a system's response to a disturbance increases the disturbance rather than damping it, such as aggressive retries adding load that causes more failures that cause more retries. Often stabilizing under normal conditions and destabilizing past a tipping point, it is a frequent latent cause of sudden cascading failures.
Latent condition
A pre-existing weakness in a system that is present and dangerous long before any incident but invisible during normal operation. It waits quietly until a trigger reveals it, and because it is a fixed property of the design, it can be hunted for in advance, unlike the unpredictable trigger.
Root-cause analysis
The structured effort to trace an outage backward from its visible symptom through a causal chain to the underlying conditions that made it possible, typically by asking 'why' repeatedly until the answers describe changeable design and organizational facts rather than individual actions.
Time-to-detect
The interval between when a failure begins and when the organization becomes aware of it. Short detection depends on monitoring the outcomes customers actually experience rather than internal metrics that can look healthy while customers cannot complete their tasks.
Time-to-recover
The interval between recognizing a failure and restoring service. It is dominated by whether the recovery procedure, such as a rollback or failover, has been rehearsed under realistic conditions, since an unexercised recovery plan is a hypothesis rather than a proven capability.
Trigger
The immediate, often mundane event that sets off an outage, such as a configuration change, a deployment, a traffic spike, or a hardware fault. Triggers are various and largely unpredictable, so their severity is governed less by the trigger itself than by the state of the system it strikes.