Run an incident with clear roles, severity levels, regular status updates and a blameless post-incident review.
Why it matters
Every system fails eventually. What separates good teams from struggling ones is not whether incidents happen but how they are handled: how fast the right people are involved, whether someone is clearly in charge, whether users and stakeholders know what is going on, and whether the team learns enough to stop the same failure happening again. Cloud engineers are usually part of an on-call rotation, so incident skills are part of the job from early on.
On-call basics
Being on call means you are the first responder for alerts during a shift, usually a week at a time. A healthy rotation has:
- Enough people: at least five or six per rotation, so each person is on call rarely enough to rest.
- A primary and a secondary responder, with clear escalation: if the primary does not acknowledge a page within a few minutes, it goes to the secondary, then to a manager.
- Handoffs: at the end of a shift, write down open issues, recent changes and anything flaky.
- Runbooks linked from every alert.
- Fair compensation and time off after a bad night. Burned-out responders make mistakes.
- A paging tool (PagerDuty, Opsgenie, incident.io, Grafana OnCall or a cloud equivalent) that handles schedules, escalation and acknowledgement.
Severity levels
Teams classify incidents so everyone knows how much urgency to apply. One common scheme:
| Severity | Example | Response |
|---|---|---|
| SEV1 | Site down or data loss for many users | All hands, incident commander, updates every 15–30 min |
| SEV2 | Major feature broken, or degraded for many users | Incident commander, updates every 30–60 min |
| SEV3 | Minor problem with a workaround | Handled in normal hours through a ticket |
When unsure, declare the higher severity. It is easy to downgrade later and costly to realize too late that you needed more help.
Incident roles
Many organizations borrow from the Incident Command System used by emergency services. In a serious incident, people take distinct roles:
- Incident commander (IC): coordinates. Decides priorities, assigns tasks, keeps everyone focused. The IC deliberately does not debug; their job is the overall picture.
- Operations lead / subject matter experts: investigate and apply fixes, and report findings to the IC.
- Communications lead: posts regular status updates to stakeholders and the public status page, so engineers are not interrupted by "any news?" messages.
- Scribe: keeps a timeline of what was seen, decided and done, with times. This becomes the backbone of the review.
In a small team one person may hold two roles, but say out loud who holds which.
The flow of an incident
- Detect: an alert fires or a user reports a problem.
- Declare: open an incident channel, set severity, name the IC.
- Mitigate first: stop the user impact before finding the root cause. Roll back the last release, turn off a feature flag, fail over, add capacity. Understanding can wait; users cannot.
- Communicate: status updates on a fixed rhythm, even when the update is "still investigating, next update at 14:30".
- Resolve: confirm the metrics are back within SLO, and announce it.
- Review: hold a post-incident review within a few days.
A useful status update has four parts:
14:05 UTC (SEV2) Checkout errors for about 20% of users since 13:42.
Impact: some payments fail; retrying may work.
Current action: rolling back release 4b1e9d0, which deployed at 13:40.
Next update: 14:25 UTC or sooner if things change.
Blameless post-incident reviews
A blameless review asks "how did our system and process allow this?", not "who messed up?" People who fear blame hide information, and hidden information makes the next incident worse. A review document usually has:
- Summary and impact (who was affected, for how long, how much error budget was used).
- A timeline from the scribe's notes.
- Contributing factors: technical (a missing test, a risky migration) and process (no canary, an alert that took 20 minutes to fire).
- What went well, as well as what went badly.
- Action items with an owner and a due date, focused on prevention and faster detection. "Be more careful" is not an action item; "add a canary stage with a 5xx stop condition" is.
Common mistakes
- Everyone debugging and nobody coordinating or communicating.
- Hunting for the root cause while users are still affected, instead of mitigating first.
- Silence: stakeholders hear nothing for an hour and assume the worst.
- Reviews that name a culprit, or that produce action items nobody owns.
- Pages without runbooks, routed to a single exhausted person.
Quick recap
- Healthy on-call has enough people, escalation, handoffs, runbooks and rest.
- Severity sets urgency; declare higher when unsure.
- Separate roles: commander, operations, communications, scribe.
- Mitigate first, update on a rhythm, and run blameless reviews with owned action items.
Try it
The Do tab is a tabletop incident drill: you break your own Compose app, run the incident with named roles and write the review.