The Follow-Through Gap: Grading Incident Reviews by What Actually Changed
Photo: RamaGaspar, CC BY-SA 4.0, via Wikimedia Commons
Most organizations now conduct incident reviews competently. The document is filed within seventy-two hours, the timeline is reconstructed with care, the tone is scrupulously even, and participants leave believing something has been learned. Seven weeks later the same service fails in the same way, and the same people gather to write the same document with a new date at the top.
That cycle is not a mystery. It is the predictable output of a process that measures completion rather than consequence.
A Ritual of Completion
Walk into almost any engineering organization and the postmortem has become a deliverable in the project sense: a template, a required field for severity, an action-item table, a distribution list. The machinery of good practice is all present. What is frequently absent is the one thing that would make the practice matter — a mechanism that forces the review to terminate in a changed system rather than a changed document.
This is not cynicism about the people involved. The engineers writing these reviews are typically the same ones who were paged at three in the morning, and they are usually sincere. The failure is structural. Reviews are scheduled close to the event, when memory is fresh and fatigue is high. Action items are drafted under time pressure and assigned to teams rather than to individuals. Nobody owns the follow-up, because follow-up is not in anyone's performance objectives. The review closes, the document enters the archive, and the underlying topology of the system remains exactly as it was.
What Blamelessness Was Meant to Buy
Blameless culture was an operational innovation before it became an etiquette. Its purpose was epistemic: to obtain accurate accounts of how complex systems actually fail by removing the incentive to hide. That is a demanding goal. It requires people to describe their own mistakes in detail, which they will only do if the account cannot be used against them.
Somewhere along the way, the principle acquired a second, softer meaning — that a review should feel safe, that no one should leave uncomfortable, that the discussion should avoid attributing outcomes to individuals. The first meaning is essential. The second, pursued without discipline, becomes a norm against naming structural problems, since structures have owners and owners have feelings. Blamelessness was supposed to buy candor. Too often it buys silence dressed as professionalism.
Three Questions Worth Grading
If reviews are to be assessed at all, they should be assessed on a small number of measurable properties. Three will do.
Action closure. Not assignment, closure. Measured at thirty and ninety days, with the count of items opened, the count completed, and the median age of the remainder. A review with fourteen open items and no closures is documentation, not remediation.
Ownership. A named person accountable for each action, with the authority to prioritize it against other work. Items assigned to “the platform team” are items assigned to no one. This is not a matter of accountability theater; it is a matter of whether anyone can answer the question “why hasn't this been done yet?” with something other than a shrug.
Design change. The number of reviews that resulted in a change to topology, contract, guardrail, or procedure — as distinct from a change to documentation or monitoring thresholds. A review that adds an alert has improved observability. A review that removes a class of failure has improved the system.
The Design-Change Test
A single question resolves most ambiguity: what can no longer happen because of this review?
If the answer is nothing — if the only change is that the team now knows to watch for this — then the review produced knowledge without capacity. That is worth something, but it is not prevention. And the distinction matters, because the most common postmortem outcome in enterprise systems is a mitigation that requires a human to notice something quickly. That is not a design fix. It is a design fix deferred to whoever happens to be awake.
Organizations that take this seriously tend to converge on a habit: every review must state, in one sentence, the specific mechanism that will now prevent recurrence, and that sentence must describe something the system does, not something a person remembers.
When Complexity Becomes the Alibi
There is a vocabulary that signals a review is about to produce nothing. “Human error” gave way, correctly, to “process gap.” Process gap gave way to “systemic complexity.” And systemic complexity now gives way, all too often, to “known limitation” — a phrase that sounds like engineering maturity and functions as a filing cabinet.
Complexity is real. Distributed systems fail in ways no single team can fully anticipate. But complexity is an explanation, not an exemption. The question a review must answer is not whether the failure was foreseeable in its specifics, but whether the organization has now made the failure less likely, less severe, or faster to contain. Any of the three counts. None of the three requires foreseeing the original event.
Publish the Closure Rate
The modest proposal is this: stop grading postmortems on attendance, thoroughness of timeline, and tone. Grade them on closure and design change, aggregate the numbers quarterly, and share them alongside availability metrics. If the closure rate is low, publish the rate — not the names.
Blamelessness protects the messenger. Follow-through protects the customer. An organization that has perfected one without building the other has not built a learning culture; it has built a very well-run ritual, and the pager will keep proving it.