Blameless postmortems: how to learn from an incident without pointing fingers
When to write a postmortem, what goes in it, and how to keep it blameless so people report problems early. Practical notes based on Google's SRE book.
Something broke, you fixed it, and everyone went back to work. That is the most common ending for an incident, and it wastes the most valuable thing the incident produced: what you now know about your system.
A postmortem is the written record that keeps that knowledge. A blameless postmortem is one people will actually write honestly. This article covers when to write one, what to put in it, and some habits that help the practice stick. The ideas come from the postmortem chapter of Google's SRE book, and I will link to it where I lean on it.
When to write one
Decide the triggers before the incident, not during it. The SRE book lists common ones:
- User-visible downtime or degradation beyond a threshold you set
- Any data loss
- An on-call engineer had to step in, for example a rollback or rerouting traffic
- Resolution took longer than a set time
- A monitoring failure, which usually means someone found the problem by hand
Any stakeholder can also ask for one. Having written criteria removes the awkward debate about whether "this one counts".
What goes in it
A good postmortem is not long. The chapter describes four ingredients:
- A record of the incident and its impact
- What was done to mitigate or resolve it
- The root causes
- Follow-up actions to prevent a repeat
Reviewers check that the key data was captured, the impact is fully described, the root cause goes deep enough, and the action items are prioritised sensibly. The chapter also says to leave out anything that could identify an end user.
Here is a skeleton you can paste into a template:
## Summary
One paragraph: what happened, how long, who was affected.
## Impact
Numbers: users, requests, money, data.
## Timeline (UTC)
- 10:02 Alert fired
- 10:09 On-call acknowledged
- 10:31 Rollback started
## Contributing causes
What made this possible? (Plural on purpose.)
## What went well / what was lucky
## Action items
| Action | Owner | Due | Priority |
Tip
Ask "what was lucky?" as well as "what went wrong?". Luck is a hidden risk: it marks the places where a small change in timing would have made the incident bigger.
What blameless means
A postmortem is blameless when it focuses on the contributing causes of an incident and does not point at a person or team. The assumption is that everyone acted with good intentions, using the information they had at that moment.
The reasoning is practical, not soft. If people expect blame, they hide problems, and you find out about the next one later and at greater cost. The chapter puts it in one line I like: "You can't 'fix' people, but you can fix systems and processes to better support people making the right choices."
It does not mean nothing matters. It means you aim your questions at the system. Compare:
- "Why did Sam push that config without checking?"
- "Why was it possible to push that config without a check, and why did nothing catch it?"
The second question leads to an action item, such as a validation step or a staged rollout. The first leads to a defensive silence.
The chapter has a small example of the same shift: an engineer who says a fragile backend keeps breaking and threatens to rewrite it alone could instead propose a rewrite as an action item, noting that the maintenance manual is long and hard to learn so future on-call engineers would benefit. The problem is still named. It is just aimed at the system.
Habits that make it stick
Writing the document is the easy part. Making it matter takes a few habits, many of them from the chapter:
- Review every draft. The chapter is blunt that a postmortem nobody reviews might as well not exist.
- Involve leaders. When senior people take part in reviews, it shows the practice is valued.
- Share widely. The book mentions a "postmortem of the month" newsletter and reading clubs that revisit older incidents.
- Keep a searchable archive. The next person who hits a similar failure should find yours in a minute.
- Practise with old incidents. The "Wheel of Misfortune" has new engineers act out a past incident, with colleagues playing the roles described in the document.
- Recognise good work. Public thanks for a well-written postmortem tells people what you value.
- Ask whether the process is too heavy. The chapter suggests surveys on whether it creates toil.
- Start gently. Try it for a trial period before making it mandatory.
Pair it with decision records
Postmortems often end in a decision: change the deploy process, drop a dependency, add a queue. Write that decision down with its reasons, as described in Architecture decision records. A postmortem tells you why you needed to change; a decision record tells you why you changed it that way.
Retries that double-charge a customer are a common cause of incidents, and the fixes are well known; Idempotency keys is a good example of a lesson that is worth building in once.
A first step for this week
Pick your last incident, even a small one. Spend thirty minutes writing the five sections above, share it with the people involved, and ask a single question: "What would have made this less likely, or easier to spot?" Put one action item on a board with an owner. That is already a postmortem culture, in miniature.
