Skip to content
Velloreth
Technology 6 min read

Incident Postmortems: Learning Without Blame

A postmortem turns an incident into a change. Learn the timeline, the root cause, the action items, and the blameless culture.

By

The purpose

To prevent a recurrence.

Not to assign blame.

A postmortem without a change is a story.

The blameless principle

A person is not the root cause.

A person acted reasonably given the information and the tools they had.

The fix is to change the system:

  • The alert
  • The guardrail
  • The documentation
  • The test
  • The interface

A blameful postmortem makes people hide information, which makes the next incident worse.

The test: does the postmortem name a person as the cause? If yes, it is not blameless.

The structure

1. Summary. What happened, in two sentences.

2. Impact. Who was affected, for how long, and how badly.

3. Timeline. The events, with timestamps.

4. Root cause. The system condition.

5. Detection. How it was found, and how long it took.

6. Response. What was done, and how long it took.

7. Action items. The changes, with an owner and a date.

8. Lessons. What was learned.

The timeline

The evidence, written from the logs.

14:02  The deploy starts
14:05  The error rate rises to 4%
14:07  The first alert fires
14:12  The on-call acknowledges
14:18  The deploy is identified as the suspect
14:24  The rollback starts
14:31  The error rate returns to normal

The timestamps are the value, because they show the detection and the response time.

The gaps are the lessons. A 5-minute gap between the alert and the acknowledgement is a finding.

The root cause

The system condition, not the trigger.

The trigger is the last event, such as a deploy.

The cause is the condition that allowed it, such as a deploy with no canary and no rollback.

The five whys is a technique to reach the condition.

Why did the site go down? A bad deploy.
Why did the bad deploy reach production? There was no canary.
Why was there no canary? The pipeline has no canary stage.
Why? It was never added.
-> The action item: add a canary stage.

The chain stops when the answer is a system condition that can be changed.

The action items

A specific change with an owner and a date.

BadGood
Improve monitoringAdd an alert on the p99 latency of checkout, Alice, Friday
Be more carefulAdd a canary stage to the pipeline, Bob, next sprint
Document the processWrite the rollback runbook, Carol, Wednesday

An action item that is not tracked is not done.

A small number of action items is better than a long list, because a long list is not completed.

The priority is the change that would have prevented the incident or shortened it.

The detection

How the incident was found.

SourceQuality
An alertgood
A customer reportpoor
A dashboard glancepoor
A random checkpoor

A customer report means the monitoring missed it, which is an action item.

The time to detect is a metric, and it is often more improvable than the time to fix.

The response

What was done, and how long it took.

The time to mitigate is the time to stop the impact, which is often a rollback.

The time to resolve is the time to fix the cause.

A long mitigation is a finding: the rollback was slow, or the runbook was missing.

The culture

A blameless postmortem requires a culture where a mistake is reported.

The signals:

  • The incident is reported early
  • The details are shared
  • The action items are completed
  • The postmortem is read

The anti-signals:

  • A person is named as the cause
  • The details are vague
  • The action items are not tracked
  • The postmortem is written to protect someone

The review meeting

The written postmortem is the artifact, and the review meeting is where it becomes a shared understanding. The meeting is short, often thirty minutes, and it is not a reading of the document, because everyone has read it. The first part walks the timeline, and the value is the questions it raises: why did the alert take five minutes, why was the rollback manual, why did nobody notice the dashboard. The second part agrees the action items, and the agreement is what makes them real, because an action item that the team did not discuss is an action item that nobody owns. The third part is the part that is often skipped: the facilitator watches for a blameful turn of phrase and redirects it, because a single “why did you deploy on a Friday” is enough to make the next incident under-reported. A postmortem that is written and never reviewed is a document, and a postmortem that is reviewed is a change.

The follow-up

The action items are the only part of a postmortem that changes anything, and the follow-up is what makes them happen. The items go into the same tracker as the other work, with an owner and a date, because a separate list is a list that is forgotten. The review at the next postmortem opens with the status of the previous items, which creates a small, regular pressure to complete them. An item that is repeatedly deferred is a signal that it is not actually a priority, and the honest response is to drop it rather than to carry it forever. The measure of a postmortem practice is not the number of documents, it is the number of incidents that did not recur, and that number is only visible if the action items are tracked to completion.

Common mistakes

A blameful tone. The information is hidden.

No timeline. The detection and the response are not measured.

A person as the root cause. The system is not changed.

A vague action item. It is not done.

No owner. Nobody does it.

No date. It is deferred forever.

Too many action items. None are completed.

A postmortem written a month later. The details are lost.

No review meeting. The lessons are not shared.

No follow-up. The action items are forgotten.

A quotable summary

A postmortem is a blameless review of an incident that records the timeline, the impact, the root cause and the action items, and its purpose is to prevent a recurrence rather than to assign blame. A person is not the root cause, because they acted reasonably given the information they had, so the fix is a system change. An action item needs an owner and a date, and a postmortem without a change is a story.

For the related topics, see incident-response-solo-developers and canary-deployments. For the broader context, ci-cd-pipeline-basics and technical-debt-conversation.

Frequently asked questions

What is a postmortem?
A written review of an incident that records the timeline, the impact, the root cause and the action items. Its purpose is to prevent a recurrence, not to assign blame. It is written soon after the incident, while the details are fresh.
Why is it blameless?
Because a person is not the root cause. A person acted reasonably given the information and the tools they had, so the fix is to change the system: the alert, the guardrail, the documentation, the test. A blameful postmortem makes people hide information, which makes the next incident worse.
What is a root cause?
The system condition that allowed the incident, not the action that triggered it. A deploy triggered the outage, and the root cause is that the deploy had no canary and no rollback. The trigger is the last event, and the cause is the condition.
What is a good action item?
A specific change with an owner and a date. "Improve monitoring" is not an action item. "Add an alert on the p99 latency of the checkout endpoint, owned by Alice, by Friday" is. An action item that is not tracked is not done.
How long should a postmortem be?
As long as it needs to be and no longer. A small incident has a short one, and a large incident has a longer one. The timeline and the action items are the parts that matter, and the prose is secondary.
When should a postmortem be written?
Within a few days, while the details are fresh and the people are available. A postmortem written a month later is a reconstruction, and the details are lost. The draft can be written during the incident, and the review held after.