Detection, Recovery, and Organizational Memory
My read is that two numbers quietly govern how much an outage hurts: how long until you notice, and how long from noticing until you're back. People obsess over preventing the trigger, which is mostly luck, and neglect these two, which are mostly craft. A system you can't see is a system that fails in the dark for an hour before anyone stirs. A system you can't recover is one where knowing the cause changes nothing. And the action items are where all of this either becomes durable memory or evaporates by Friday. An outage you don't turn into a changed system is just an expense. Turn it into a fix with an owner and you finally bought the intelligence you paid for.
Knowledge check
Two organizations suffer outages from an identical cause. One is back in fifteen minutes; the other takes six hours. Both eventually found the same root cause. What most likely differed between them, in terms of the two key time metrics?
Show the answer
Their time-to-detect and time-to-recover, not their root cause. The faster organization likely had monitoring that surfaced the problem quickly (short time-to-detect) and a rehearsed, working recovery path such as a fast rollback or failover (short time-to-recover). The slower one may have discovered the problem late and then discovered that its recovery procedure was untested and did not work under real conditions.