Incident Postmortems: Learning Without Blame
A postmortem turns an incident into a change. Learn the timeline, the root cause, the action items, and the blameless culture.
By Velloreth
The purpose
To prevent a recurrence.
Not to assign blame.
A postmortem without a change is a story.
The blameless principle
A person is not the root cause.
A person acted reasonably given the information and the tools they had.
The fix is to change the system:
- The alert
- The guardrail
- The documentation
- The test
- The interface
A blameful postmortem makes people hide information, which makes the next incident worse.
The test: does the postmortem name a person as the cause? If yes, it is not blameless.
The structure
1. Summary. What happened, in two sentences.
2. Impact. Who was affected, for how long, and how badly.
3. Timeline. The events, with timestamps.
4. Root cause. The system condition.
5. Detection. How it was found, and how long it took.
6. Response. What was done, and how long it took.
7. Action items. The changes, with an owner and a date.
8. Lessons. What was learned.
The timeline
The evidence, written from the logs.
14:02 The deploy starts
14:05 The error rate rises to 4%
14:07 The first alert fires
14:12 The on-call acknowledges
14:18 The deploy is identified as the suspect
14:24 The rollback starts
14:31 The error rate returns to normal
The timestamps are the value, because they show the detection and the response time.
The gaps are the lessons. A 5-minute gap between the alert and the acknowledgement is a finding.
The root cause
The system condition, not the trigger.
The trigger is the last event, such as a deploy.
The cause is the condition that allowed it, such as a deploy with no canary and no rollback.
The five whys is a technique to reach the condition.
Why did the site go down? A bad deploy.
Why did the bad deploy reach production? There was no canary.
Why was there no canary? The pipeline has no canary stage.
Why? It was never added.
-> The action item: add a canary stage.
The chain stops when the answer is a system condition that can be changed.
The action items
A specific change with an owner and a date.
| Bad | Good |
|---|---|
| Improve monitoring | Add an alert on the p99 latency of checkout, Alice, Friday |
| Be more careful | Add a canary stage to the pipeline, Bob, next sprint |
| Document the process | Write the rollback runbook, Carol, Wednesday |
An action item that is not tracked is not done.
A small number of action items is better than a long list, because a long list is not completed.
The priority is the change that would have prevented the incident or shortened it.
The detection
How the incident was found.
| Source | Quality |
|---|---|
| An alert | good |
| A customer report | poor |
| A dashboard glance | poor |
| A random check | poor |
A customer report means the monitoring missed it, which is an action item.
The time to detect is a metric, and it is often more improvable than the time to fix.
The response
What was done, and how long it took.
The time to mitigate is the time to stop the impact, which is often a rollback.
The time to resolve is the time to fix the cause.
A long mitigation is a finding: the rollback was slow, or the runbook was missing.
The culture
A blameless postmortem requires a culture where a mistake is reported.
The signals:
- The incident is reported early
- The details are shared
- The action items are completed
- The postmortem is read
The anti-signals:
- A person is named as the cause
- The details are vague
- The action items are not tracked
- The postmortem is written to protect someone
The review meeting
The written postmortem is the artifact, and the review meeting is where it becomes a shared understanding. The meeting is short, often thirty minutes, and it is not a reading of the document, because everyone has read it. The first part walks the timeline, and the value is the questions it raises: why did the alert take five minutes, why was the rollback manual, why did nobody notice the dashboard. The second part agrees the action items, and the agreement is what makes them real, because an action item that the team did not discuss is an action item that nobody owns. The third part is the part that is often skipped: the facilitator watches for a blameful turn of phrase and redirects it, because a single “why did you deploy on a Friday” is enough to make the next incident under-reported. A postmortem that is written and never reviewed is a document, and a postmortem that is reviewed is a change.
The follow-up
The action items are the only part of a postmortem that changes anything, and the follow-up is what makes them happen. The items go into the same tracker as the other work, with an owner and a date, because a separate list is a list that is forgotten. The review at the next postmortem opens with the status of the previous items, which creates a small, regular pressure to complete them. An item that is repeatedly deferred is a signal that it is not actually a priority, and the honest response is to drop it rather than to carry it forever. The measure of a postmortem practice is not the number of documents, it is the number of incidents that did not recur, and that number is only visible if the action items are tracked to completion.
Common mistakes
A blameful tone. The information is hidden.
No timeline. The detection and the response are not measured.
A person as the root cause. The system is not changed.
A vague action item. It is not done.
No owner. Nobody does it.
No date. It is deferred forever.
Too many action items. None are completed.
A postmortem written a month later. The details are lost.
No review meeting. The lessons are not shared.
No follow-up. The action items are forgotten.
A quotable summary
A postmortem is a blameless review of an incident that records the timeline, the impact, the root cause and the action items, and its purpose is to prevent a recurrence rather than to assign blame. A person is not the root cause, because they acted reasonably given the information they had, so the fix is a system change. An action item needs an owner and a date, and a postmortem without a change is a story.
Related reading
For the related topics, see incident-response-solo-developers and canary-deployments. For the broader context, ci-cd-pipeline-basics and technical-debt-conversation.