Incidents and postmortems

Why it matters

Every team that runs software has incidents: an outage, a bad deploy, a full disk, a payment provider that stops responding. What separates mature teams is not that they never have incidents but how they handle them: they restore service quickly, communicate clearly while it is happening, and learn from it afterwards so the same failure does not happen again. The learning step is the postmortem (also called an incident review or retrospective), and the most effective kind is blameless.

During the incident

An incident is any unplanned event that harms users or puts them at risk. Most teams use severity levels, for example:

A calm response follows a few steps:

1. Detect. Ideally an alert on error rate or latency fires before users notice. Otherwise a support ticket or a teammate reports it.
2. Declare and assign roles. Someone says "this is an incident" and opens a channel. One person is the incident commander, who coordinates and makes decisions but does not debug. Others investigate. Someone handles communication with users and stakeholders.
3. Mitigate first. Restore service before finding the root cause. Roll back the last deploy, turn off a feature flag, scale up, or fail over. A rollback in five minutes beats a perfect fix in two hours.
4. Communicate on a schedule. Short updates at regular intervals ("Checkout errors since 14:05. We have rolled back the 14:00 deploy and errors are falling. Next update 14:45."), even when there is no news.
5. Keep a timeline. Note the time of every observation and action in the incident channel. You will need it for the postmortem.
6. Resolve when users are no longer affected, then schedule the review.

The blameless postmortem

A postmortem is a written review of what happened, why, and what will change. Blameless means it assumes everyone acted reasonably given the information, tools and pressure they had at the time. The question is never "who made the mistake?" but "how did our system make this mistake easy to make, and hard to catch?"

This is not about being nice. It is practical. If people are punished for mistakes, they hide them, and the organisation stops learning. If an engineer can run one command that deletes production data, the real problem is the system that allowed it, and the fix is a safeguard, not a reprimand.

A typical postmortem has these sections:

# Postmortem: Checkout errors, 2026-10-23

**Severity:** SEV2 · **Duration:** 14:05–14:41 UTC (36 min) · **Status:** Action items open

## Summary
A deploy changed the tax service URL. 22% of checkouts failed for 36 minutes.

## Impact
~1,300 failed checkout attempts; ~400 users affected; no data loss.

## Timeline (UTC)
- 14:00 Deploy 2026.10.23-2 released
- 14:05 Error-rate alert fires for /checkout
- 14:12 Incident declared; Ana is incident commander
- 14:31 Rollback started
- 14:41 Error rate back to normal; incident resolved

## Root cause and contributing factors
- The tax service URL was moved into a new config variable, set in staging but not production.
- The app started without it and only failed when checkout called the tax service.
- The deploy checklist has no step for new config variables.

## What went well
- The alert fired within 5 minutes.
## What went poorly
- Rollback took 19 minutes to start because the runbook was out of date.

## Action items
| Action | Owner | Due |
|---|---|---|
| Fail at startup if required config is missing | Ravi | 2026-10-30 |
| Add config check to the deploy pipeline | Mei | 2026-11-06 |
| Update rollback runbook and practise it | Ana | 2026-11-06 |

Notes on writing a good one:

• Contributing factors, plural. Real incidents rarely have one cause. Asking "why?" several times in a row (the 5 Whys) helps you move from "a variable was missing" to "nothing checks for missing variables".
• Action items are specific, owned and dated. "Be more careful" is not an action item. "Fail at startup if a required variable is missing" is.
• Prefer fixes that prevent or detect the whole class of problem over fixes for this one instance.
• Describe actions neutrally. Write "the deploy was released at 14:00", not "Ravi carelessly deployed".
• Share it widely and review it in a meeting. Track action items until they are done; a postmortem whose actions never happen is just a story.

Common mistakes

• Debugging for an hour before trying a rollback.
• Everyone investigating and nobody coordinating or communicating.
• Postmortems that name and shame, or stop at "human error".
• Action items with no owner or due date, or so many that none get done.
• Skipping the postmortem for "small" incidents that keep repeating.

Recap

• Detect, declare, assign roles, mitigate first, communicate on a schedule, keep a timeline.
• Blameless postmortems ask how the system allowed the failure, not who caused it.
• Summary, impact, timeline, contributing factors, what went well and poorly, and owned, dated action items.

Try it

In the Do tab, run a short incident simulation and write a blameless postmortem for it.