The alert fires at 02:47. By half past four the service is back, the backlog has drained, and someone posts a short message in the team channel: "Fixed. Rolled back the config." Everyone goes back to bed.

That message is not a post-mortem. It is a sigh of relief. And relief is a poor teacher. Without a written review, the same failure returns in six weeks wearing slightly different clothes, because nothing about the system or the process that produced it has changed.

Start with the timeline, not the cause

Write the timeline before you write any conclusions. Conclusions drawn from memory alone tend to be tidy, flattering and wrong. Pull the facts from places that were paying attention: alert history, deploy logs, chat timestamps, ticket comments, the on-call rota, and the notes whoever was holding the pager scribbled at the time.

Reconstruct events minute by minute, and record what people believed at each point, not what you know now. That distinction is where most of the value sits.

1. 02:47 — first alert: payment authorisation error rate above 5%
2. 02:52 — on-call acknowledges, checks the dashboard, sees a spike across all regions
3. 03:05 — working assumption is an upstream provider problem; the status page is checked twice
4. 03:38 — a saturated database connection pool is spotted on the payments cluster
5. 04:05 — rollback of the 02:30 configuration change begins
6. 04:22 — error rate returns to baseline; backlog drain continues until 05:10

Aim for dull precision. "03:38 — pool saturation found" tells you more than three paragraphs of narrative about mounting pressure.

What blameless actually means

Blameless does not mean vague. You can name an action and its consequence without attaching it to someone's competence. "The configuration was changed without a staging check" is a factual statement. "Someone was careless" is a verdict, and verdicts shut down the conversation you need to have.

The reason is practical rather than sentimental. If a single person's ordinary mistake can take down production, you have a system that permits it. Blaming them fixes nothing and guarantees you hear less next time.

At 03:10 we believed the fault lay with the upstream provider, so twenty minutes went into checking their status. That was a reasonable reading of the evidence available at the time.

Blameless does not mean unaccountable, either. Action items still have owners and dates. The difference is that accountability points forwards, at the work, rather than backwards, at a person.

Dig past the first plausible cause

The first cause you find is usually a trigger, not a root cause. A configuration change triggered the outage; the reason it could reach production unchallenged is the thing worth fixing. Three questions get you most of the way there.

Why was the failure possible?

What assumption turned out to be false? Perhaps the pool size had been sized for the old traffic profile, or the config schema allowed a value that the application could not handle.

Why was it not caught earlier?

Tests, staging, canaries, reviews, dashboards. If a check exists but did not fire, that gap is a finding in itself. If no check exists, that is a finding too.

Why did recovery take as long as it did?

Twenty minutes of misdirected effort is not a failure of intelligence, it is a gap in observability or runbooks. If one signal had pointed at the database pool immediately, the incident would have been shorter.

List the contributing factors alongside the root cause. Real incidents rarely have one.

Turn findings into action items that survive

Most post-mortems fail here, not in the analysis. A list of eleven worthy tasks is abandoned by the following Tuesday. Three or four that actually ship are worth far more.

• One owner, one ticket, one date. "The team will look at this" is not an action item.
• Small enough to finish inside a sprint. If it is not, split it.
• Do not write "be more careful". It cannot be verified, cannot be closed, and changes no behaviour.
• Say what done looks like. "Add a canary deploy check that halts the rollout on a 2% error-rate increase" beats "improve deployment safety".
• Spread the load. Prevention, detection, mitigation, response — pick at least one from a category you have been neglecting.

Then put the items in the same tracker as all other work, with a routine review of open ones. An action item that lives only inside a document is a wish.

Write it so people actually read it

Structure beats prose. Put a short summary at the top: what broke, who or what was affected, for how long, and what you are changing. Then the impact in numbers, the timeline, the contributing factors, what went well, and the action items.

Keep the main document to a page or two and push raw logs, graphs and transcripts into an appendix. Write in plain language. Reviewers outside the incident should be able to follow it without asking what "the pool" means. And note what went right — a fast rollback, a clear handover, a useful alert — because those are behaviours worth repeating.

Run the review meeting properly

Book it within a few days, while memories are still usable. Forty-five minutes is usually enough. Invite the responders, the people who build the affected systems, and anyone who supported the incident from outside the team.

Walk the timeline in order rather than jumping to the fix. Ask questions that invite detail: "What did the dashboard show at that point?", "What would have helped you decide faster?" If the conversation turns towards who did what, redirect it to what the system allowed, and what would have made the right action obvious. Rotate the facilitator so the review does not become one person's ritual.

Make the next incident cheaper

A good post-mortem passes a simple test: someone who was not there can read it and know what to do differently. That is the whole point. You will not prevent every outage, but you can make the next one shorter, quieter and less dependent on luck. Write it down, give the actions real owners, and check in on them next month — that is what turns a bad night into something useful.

Photo: Yan Krukau / Pexels