Testing Well: Realism, Blamelessness, and Closing the Loop

Running a test is not the same as running a test well, and a badly run test can be worse than none — it manufactures false confidence, the most dangerous asset in a recovery programme. Three disciplines separate a test that produces knowledge from a test that produces reassurance.

The first is realism, and its chief enemy is the temptation to help the test pass. A test that has been quietly smoothed — the right expert on standby, the systems pre-warmed, the tricky step skipped 'because we know it works' — measures a recovery that will never happen, because the real disaster will not arrive during business hours with the right expert rested and available. The whole value of a test is in the friction it reproduces. So the realistic test is deliberately inconvenient: it assumes the key person is unreachable, so someone else must run the runbook cold. It withholds the tribal knowledge, so the documentation has to carry the weight it will actually have to carry at three in the morning. The most instructive drills are the ones where the usual expert sits silently on their hands and watches whether the plan works without them, because in the real event they may be on a plane, asleep, or gone. A test you have to help pass has already told you its answer; you simply have to be honest enough to hear it.

The second discipline is blamelessness, and it is the one organisations most often get wrong. A recovery test exists to find failures — that is its entire function — and the failures it finds will frequently be someone's mistake: an out-of-date runbook, a backup nobody verified, an access grant that was never renewed. If finding those failures gets people blamed, two things happen, both fatal. People stop volunteering the failures they notice, and the tests quietly become performances designed to pass rather than investigations designed to learn. The only way to keep tests honest is to make it unambiguously safe to fail one. The failure belongs to the plan, not the person; the person who surfaced it did you a favour by finding it in rehearsal. This is the practical meaning of treating failure as intelligence: intelligence is only cheap if the people who gather it are rewarded rather than punished for what they find. A blameless test costs you an uncomfortable finding. A blameful test costs you the truth.

The third discipline is closing the loop, which is where most testing programmes silently fail. A test that surfaces problems and then files a report that no one acts on has spent real money to buy intelligence and then thrown the intelligence away. Every finding must become a specific, owned, tracked action: this runbook step gets corrected, this access gets automated, this dependency gets documented, and someone's name is attached to each with a date. And then — the part that requires institutional stamina — the next test verifies that the fix actually took. Recovery capabilities decay continuously, because the systems they protect change continuously; a fix confirmed once is a fix that will quietly rot until the next test finds it broken again. This is why testing is a practice and not a project. The plan you validated last year describes systems that have since been patched, rewired, and rebuilt around it. Only a repeating cycle of test, fix, and re-test keeps the plan in contact with the reality it claims to protect.

There is a cultural payoff to all of this that is easy to miss and worth stating plainly. An organisation that tests its recovery regularly, realistically, and blamelessly is not merely one with better-tested plans. It is one where failure has been domesticated — turned from a catastrophe that happens to you into an experiment you run on purpose. The teams that fail over every month are calm during real failovers, because they have made the extraordinary ordinary. They have internalised the domain's whole thesis: that failure is not the enemy of reliability but its cheapest teacher, and that the disciplined thing to do with a teacher that cheap is to visit it often, on your own schedule, and take careful notes. The alternative is to wait for the disaster to run the test for you — unannounced, at the worst hour, graded on a curve you did not set, with the intelligence delivered at the highest possible price. Everything in this course reduces to a single preference: buy the intelligence early and cheaply, or have it sold to you late and dear. The tested plan is simply the cheaper purchase.