Incident Review Without Looking for Someone to Blame

Most incident reviews conclude with a variant of "someone made a mistake, we will be more careful." That conclusion is always available, always true in a narrow sense, and changes nothing — because the next person will make a different mistake under the same conditions.

For a separate people-operations perspective, the explanatory note covers separating accountability from responsibility.

Blameless review is not politeness. It is a technique for getting information you cannot otherwise obtain, and the mechanism is worth understanding before the process.

Why blame destroys the data

People stop volunteering information. The person who knows exactly what happened is the person most exposed by saying so. In a blaming culture, that account becomes careful, partial, and late — and it was the most valuable input available.

"Human error" ends the investigation at the most interesting point. Someone ran the wrong command. Why was that command available? Why did it not require confirmation? Why did the safe and the dangerous version look nearly identical? Why was the person doing this at 2am? Those questions have actionable answers; "be careful" does not.

Hindsight distorts everything. After an outage the correct action is obvious. It was not obvious at the time, with the information available at the time, under time pressure. The question is not why they did the wrong thing — it is why it looked like the right thing. That reframing is the whole method.

Running one

Within a few days, while memory is fresh and before the account has been smoothed into a story.

Include the people who were there, and make it explicit that the goal is understanding rather than accountability.

Build the timeline first, from evidence. Logs, alerts, deploys, chat messages, with timestamps. Do this before discussing causes — the discussion changes what people remember.

Separate what was known at each moment from what is known now. The timeline should record what the responder could see, not what was true. If a dashboard showed misleading data, that is a finding.

Ask how, not why. "Why did you restart it?" invites justification. "How did the situation appear to you at that point?" invites description.

Questions that produce findings

How did the responder learn about it? If it was a customer, monitoring is a finding.

What did they check first, and what did it tell them? Misleading signals are a finding.

What made diagnosis slow? Missing logs, a broken trace chain, a dashboard that did not exist, a runbook that was wrong.

What made the fix risky or slow? Deployment time, lack of a rollback path, needing someone unavailable.

What made this possible? Not who — what. A configuration that could be changed without review, a limit with no alert, a dependency with no timeout.

What made it worse than it needed to be? Retry storms, a cascade, an alert that fired thousands of times.

What went right? Genuinely useful, and it identifies what to protect. Someone noticed early, a circuit breaker held, a rollback worked.

And the counterfactual question worth asking last: what would have had to be different for this to be caught before production? The answer names a control, not a person.

Root cause is the wrong shape

Serious incidents rarely have one cause. They have a set of conditions that were individually tolerable.

A configuration change was possible without review, and the staging environment did not have the same data volume, and the alert threshold was set from a period of lower traffic, and the on-call engineer had not seen this system before, and the rollback required a deploy that took twelve minutes.

Remove any one and the incident is smaller. Naming one as the root cause discards the other four, which are the ones you can also fix.

Write contributing conditions, plural. Then decide which are worth addressing.

Actions that are actually actions

Most incident reviews produce a list nobody completes, and the reason is usually visible in the wording.

Bad actions: "be more careful", "improve monitoring", "add more tests", "review the process", "document this better". Unassignable, uncheckable, and never done.

Good actions name a specific change, an owner and a date. "Add a lock timeout to the migration runner — Anna — by the 14th." "Alert when queue depth exceeds 1000 — Dmitri — by the 20th."

Fewer is better. Three actions that are completed beat twelve that are filed. A review generating twelve is usually one that failed to prioritise.

Prefer changes that remove the possibility over changes that add vigilance. A confirmation prompt is weaker than making the dangerous operation require a different tool. A checklist is weaker than an automated check. Vigilance decays; constraints do not.

Track them, and report completion. Actions from reviews that are never completed teach everyone that the review is theatre — and then the accounts get thinner, and you are back to blame by another route.

The write-up

Short, and written for someone who was not there.

What happened, in a paragraph.

Impact — who was affected, for how long, in concrete terms. "Checkout unavailable for 23 minutes, approximately 400 failed orders" rather than "degraded service".

Timeline, from the evidence.

Contributing conditions.

What went well.

Actions, with owners and dates.

Publish it internally. The value is in people who were not involved learning what the system does under stress. An incident review read by three people has produced a fraction of its value.

Two organisational things that decide whether this works

Leadership behaviour in the first hour. If a senior person asks "who deployed it" before asking "what broke", the culture is set regardless of what the policy document says.

Reviewing near-misses too. The incident that was caught before customers noticed contains the same information at a fraction of the cost, and reviewing those is what turns a reactive process into a preventive one.

The summary

Human error is where the investigation should start, not stop. Ask what made the wrong action look right.

Incidents have conditions, not a cause. List them all; fix the ones worth fixing.

Actions need an owner and a date, and there should be few of them. Prefer removing the possibility over asking for more care.

Publish it, and complete the actions — or the next review will get you a thinner account than this one did.