Incidents, On-Call and Blameless Post-Mortems

The page has fired. Somebody is awake. What happens next decides if the incident lasts 10 minutes or 3 hours. In most bad incidents the code is not the main problem. The main problem is people: five engineers each debugging the same thing in different chat rooms, nobody telling the customers anything, a risky fix tried at random, and nobody sure who is in charge.

This lesson is about the human side of reliability. We cover what an incident is, how to give it a severity, the roles people take, what to do in the first hour, how to talk to others, how on-call should work so it does not burn people out, and how to write a post-mortem that teaches without blaming. At the end there is a complete fictional post-mortem you can copy as a template.

This is lesson 6 of a nine-lesson series. The page that starts an incident usually comes from the alerts in Alerting and Dashboards That People Trust, and the numbers we talk about (error budget, burn) come from SLIs, SLOs and Error Budgets. The next lesson, Reliability Patterns, turns the cause in our example post-mortem (a retry storm) into running code. If you want the general habit of finding root causes, see also the debugging series, for example the debugging lessons.

Words used in this lesson

• An incident is an event that hurts, or may soon hurt, users or the business and needs a coordinated human response.
• Severity (SEV) is a label for how bad an incident is. A lower number usually means worse (SEV-1 is the most serious).
• Mitigation means making the pain stop or shrink (a rollback, turning off a feature), even before you know the full cause.
• Resolution means the real problem is fixed and the service is healthy again.
• Incident Commander (IC) is the one person who runs the incident: keeps everyone coordinated, decides what to try, and speaks for the team. The IC does not usually fix things personally.
• Deputy is the IC's helper and hot standby.
• Scribe is the person who writes down what happens and when.
• Subject matter expert (SME) is someone who knows a specific system well and works on the fix.
• Communications liaison (customer liaison, internal liaison) is the person who writes updates for customers or for the rest of the company.
• On-call means being the person who will answer the page during a set period.
• Escalation means calling someone else (a second person, a manager, a specialist) when you cannot solve it alone or fast enough.
• A post-mortem (also called a retrospective or incident review) is a document and a meeting after an incident: what happened, why, and what to change.
• Blameless means the review looks at systems and decisions, not at who to punish.
• Contributing factor is one of several conditions that together allowed the incident. Most incidents have many. "Root cause" suggests there is only one, so I prefer this term.
• Action item is a concrete follow-up task with an owner and a due date.
• MTTD, MTTA, MTTM, MTTR are mean time to detect, acknowledge, mitigate and recover/resolve. We see below why the word "mean" can fool you.

What counts as an incident

Not every bug is an incident. An incident is when something is going wrong now (or is about to) and one person alone cannot handle it. A useful working definition: users are hurt, or the error budget is burning fast, or data is at risk, and several people need to coordinate.

The big benefit of deciding "yes, this is an incident" quickly is that it switches the team into a different mode: one channel, roles, regular updates. The cost of calling it too early is small (a few people join and leave). The cost of calling it too late is big. So the habit is: when in doubt, declare it. You can always downgrade.

Severity levels

A severity level tells everyone how serious the situation is, and how much effort and risk are justified. The PagerDuty documentation uses SEV levels where a lower number is more urgent, and says anything above SEV-3 is a major incident that gets a more intensive response. Their example meanings, in my words:

Your levels will differ, and that is fine. What matters is that they are written down, short, and based on user impact, not on how scary the technology sounds.

Two habits from that documentation that I like a lot:

1. If you are unsure which of two levels it is, treat it as the higher one. During an incident is not the time to argue about the level. You can discuss it in the post-mortem. Raising the level costs a little. Lowering it too early costs a lot.
2. A higher severity allows riskier fixes. For a SEV-1 you may do things (a quick rollback in the middle of the day, a restart of the main database proxy) that you would never do for a SEV-3.

Roles: why not everybody just fixes it

Imagine ten people in one room, all shouting ideas, and two of them typing commands on the same server. That is how a small incident becomes a big one. The fix is to give people roles, so each person has a clear job. The PagerDuty roles page describes these, which I summarise:

The same page makes a point I want to repeat: the structure is flexible. On a small incident one person may be IC, scribe and liaison. The idea is not "ten roles for every incident". The idea is that each of the jobs gets done, and that for any job, everyone knows who is doing it. For a small team of four, a good minimum is: one IC, one person fixing, and one person who writes down what happens and talks to others.

The Incident Commander is not the best engineer

This surprises people. The IC is often not the person who knows the system best. The IC's skill is staying calm, asking good questions ("what changed in the last hour?"), keeping the group focused, and making a decision when people disagree. Their power comes from the role, not from rank. A junior engineer who is IC can tell the CTO, "please take that to the side thread". Having people trained in this role before the bad night is the best preparation.

If you join an incident and nobody is IC, ask who is, and if no one is, say "I'll take IC" and announce it. Taking charge for a few minutes is far better than leaving a hole.

The first hour

A good first hour has a rhythm. This is my own checklist, built from the "during an incident" guidance I read and from common practice:

Minute 0 to 5: stabilise the team

1. Acknowledge the page. The sender must know that a human is on it.
2. Check it is real. Open the dashboard from the alert. Is the SLO page true? (A quick look, not a long study.)
3. Declare an incident and give it a severity. Open one channel (a chat room, plus a call if needed). Everyone goes to that one place.
4. Say who is IC. If you are alone, you are IC for now. Say it out loud in the channel.

Minute 5 to 20: find a safe action, not the final answer

5. Ask: what changed? Deploys, config changes, feature flag flips, traffic spikes, a dependency failing. Most incidents follow a change.
6. Pick the quickest safe mitigation. Roll back the deploy. Turn off the feature flag. Shift traffic away from a bad zone. Add capacity. Restart a stuck process. A good rule that matches the PagerDuty guidance: for a bad deployment, roll it back; for a stuck application, do a rolling restart; for a flood of events, check automatic throttling.
7. Delegate each action to a named person, one at a time, and have them report back ("Sam: roll back checkout to v2.40, tell me when pods are healthy").
8. Write it down (the scribe). Time, what, who.

Minute 20 and later: communicate and confirm

9. Update stakeholders regularly, even when there is no news ("still investigating, next update in 15 minutes"). Silence makes people panic and interrupt.
10. Confirm recovery with the SLI, not with a feeling. The error ratio and latency must go back to normal and stay there. Do not close the incident the moment the chart dips. Wait a short, agreed time (10 minutes is common).
11. Declare it resolved, say what the follow-up is (a post-mortem will come), and thank people.

Mitigate first, understand later

The single most important idea of incident response: stopping the harm comes before finding the cause. If rolling back the last deploy restores service, do it, even if you have no idea why the deploy was bad. You can study the bad build later, in calm daylight, with all the logs, because you kept the evidence. Debugging a live fire for an hour while users suffer is a common mistake.

The rollback must be a safe habit. That is why release engineering matters: a rollback that works in two minutes is the best incident tool there is.

Things that waste incident time

• Many channels. Two rooms means two versions of the truth.
• Too many people talking. Let the SMEs talk, and others listen. Anyone can join to watch, but only those with a job should speak up.
• Debating the severity or the cause while the system is down.
• The "hero" who goes quiet and works alone for 40 minutes. Ask for a status every 10 or 15 minutes.
• No handoff. An incident that runs longer than 2 to 4 hours needs people to rotate. A tired person is a slow, error-prone person. The IC role itself should hand over, with a clear spoken handoff, "I'm handing IC to Priya. Current state is...".

Communication during the incident

Customers and colleagues need three things: what we know, what we're doing, and when we'll tell you more. A short template that works for a status page post:

[Investigating] Some checkout requests are failing. We are looking into it.Next update in 30 minutes.[Identified] We found a change that caused failures and are rolling it back.Next update in 20 minutes.[Resolved] Checkout is working normally. Some payments made between 14:09 and 14:46 UTCmay have failed. No card was charged for a failed order. A full report will follow.

Rules for such messages (matching the PagerDuty customer guidance I read, in my words):

• Be honest and plain. Don't guess the cause in public.
• Avoid the word "outage" unless it really was a full outage. "Incident" or "degraded service" is more accurate for partial problems.
• Give a time for the next update, and keep it, even if the update is "no news".
• One person (the customer liaison) posts, after the IC approves.
• Mention real customer impact in terms customers understand (what they could not do, for how long).

Inside the company, a short summary every 30 minutes for stakeholders keeps managers from joining the call to ask questions.

On-call that does not burn people out

On-call means someone is always responsible for answering the page. It can be healthy or it can destroy a team. Here are the habits that make the difference.

Structure

• A primary and a secondary. If the primary doesn't acknowledge in a few minutes (5 is common), the page goes to the secondary automatically. Nobody should be alone with no backup.
• A clear rotation, usually one week per person, published in advance. People can swap shifts and plan their lives.
• Not too small. One person on call every other week is burnout. Aim for a rotation large enough that each person is on call a few weeks a quarter. If your team is too small, borrow people from a nearby team.
• A written handoff at the end of every shift: open issues, flaky alerts, things to watch.
• Rest after a bad night. If someone was paged at 3 a.m. for two hours, they don't start work at 9. Make it a rule and not a favour.
• Pay or time off for on-call duty. It is real work and real disruption.

Habits for the person on call

• Carry the laptop and a charged phone. Know the routes to the runbooks and dashboards before you need them.
• Test that your page actually reaches you (including "do not disturb" settings).
• Escalate early. Asking for help in the first 10 minutes costs little. Waiting 90 minutes to avoid "bothering" someone costs a lot. In PagerDuty's guidance for the call, the advice about the incident commander is "never hesitate to page the IC. It's much better to have them and not need them." A good team treats a page that wasn't needed as a normal, forgiven event.
• After every shift, fix one thing: improve a runbook, tune a noisy alert, write a small automation. On-call gets better only if the people doing it improve it.

Health numbers for a rotation

Track these every quarter: pages per shift, percent of pages that were actionable (we saw 3 of 48 in the last lesson, which is a team in trouble), pages outside working hours, and how long the average incident took. If the numbers are bad, fix the alerts, not the people.

Time measures: what to track, and a trap

I wrote incident-times.mjs to compute the usual measures from a timeline. The timeline is the one from the fictional post-mortem later in this lesson. Output:

time to detect  (start -> detected):    8 mintime to ack     (detected -> ack):      2 mintime to mitigate (start -> mitigated):  37 mintime to resolve (start -> resolved):    49 minuser impact window (start -> mitigated): 37 minequivalent full-outage minutes: 14.1 of 43.2 => 32.5% of the 30-day budget

Read each line:

• Time to detect (8 min): from the first failing request to the page. Lower is better, and it is mostly an alerting question.
• Time to acknowledge (2 min): from page to a human taking it.
• Time to mitigate (37 min): from when users started suffering to when the pain stopped. This is the number users feel. It is the one to reduce.
• Time to resolve (49 min): from start to "all done and confirmed".
• Equivalent full-outage minutes: the impact (38% of requests failing for 37 minutes) is like 14.1 minutes of total outage, out of a monthly allowance of 43.2 minutes at 99.9%. So this one incident used about a third of the monthly error budget. That is the number to bring to the planning meeting.

Real incidents have a pattern

Before the fictional example, a real-world note. The public list of post-mortems in the danluu/post-mortems repository is a great way to see how real failures look. I only read the one- or two-sentence summaries in its README, not the full reports behind each link, so I summarise only what the summaries say.

A few that touch this series' themes:

• A summary of an Amazon storage incident says an operator typed a command with a typo while following an established playbook, and it removed far more servers than intended. The lesson often drawn from that kind of story is that tools should limit the blast radius of a command, not that the person should be more careful.
• A summary of an older Amazon incident says a network disruption raised load on a metadata service, response times exceeded timeouts, storage nodes took themselves down, and nodes that were down kept retrying, so the load could not fall. That is a retry-amplification story, which is the topic of the next lesson.
• A summary of a GitHub incident mentions an inefficient endpoint called by a client that retried on timeout. Again retries amplified the problem.
• A summary of the crates.io incident (the Rust package registry) says a bad deployment broke download URLs for about 13 minutes, and the failed requests included retry attempts from cargo, the client tool.

I did not follow those links, so I can't tell you the details or confirm the summaries. I mention them because they show that retries and timeouts show up again and again in real post-mortems, which is why the next lesson is about them. The repository's README, which I read, is an index of reports made by many different companies, and it groups them under headings such as config errors, hardware, conflicts, time and database. It's a good place to read three or four full reports and notice how the best ones are written.

Blameless post-mortems

After the incident, the team writes a document and holds a short meeting. The goal is learning. The PagerDuty process page says it in one sentence that I like: a post-mortem should be a "blame-free, detailed description of exactly what went wrong", with steps to stop it happening again, and it should also cover how the incident response itself went.

What "blameless" means (and does not mean)

It does not mean "nobody did anything wrong" or "no accountability". It means we start from the belief that people were doing their best with the information and tools they had at that moment, and that a system that lets one person's single mistake cause a big outage is a system problem. If we punish the person, three bad things happen: people hide mistakes, people stop reporting near-misses, and the same system weakness stays in place for the next person.

Compare two sentences about the same event:

The blameless versions name people by role where needed, describe conditions, and lead directly to action items. You can still say who did what in the timeline (it is just facts) without any judgement.

A useful question to replace "who did this?" is "what made this seem like a reasonable thing to do at the time?" The answer is nearly always something about the system: a confusing name, a missing guard, a doc that was out of date.

Contributing factors, not a single root cause

Almost no serious incident has one cause. There is the trigger (the thing that started it), and there are the conditions that let the trigger hurt (no limits, no tests, no alert, no runbook). Writing "root cause: human error" is useless, because it ends the thinking exactly where learning should begin. List several contributing factors and fix as many as are worth the cost. A popular tool is the "five whys": keep asking "why did that happen?" until you get to something you can change in the system. Use it as a thinking aid, and be careful not to force a single chain when the truth is a web.

The process (adapted from the PagerDuty process page)

1. The IC names an owner for the post-mortem right after the incident. The owner is not necessarily the person who fixed it.
2. Schedule the meeting first, then write. PagerDuty's rule: within 3 calendar days for a SEV-1 and 5 business days for a SEV-2. Memory fades quickly, so don't wait weeks.
3. Start with the timeline. Use the scribe's notes. For each item, link to the data: a graph, a log search, a message. If you can't link evidence, mark the line as "from memory".
4. Analyse. Capture the impact (numbers!), what happened, and why.
5. Share a draft a day before the meeting, so people can comment in advance and the meeting is short.
6. Hold a short meeting (15 to 30 minutes in PagerDuty's description) to agree on the facts and the actions. It's not for debate about guilt.
7. Create action items as tickets with owners and dates. PagerDuty's advice is to avoid creating too many; keep to the ones that really must be done. A list of 30 actions guarantees none get done.
8. Share the lessons with the wider company and keep the document where people can find it.
9. Follow up. Check weeks later that the actions were done. A post-mortem whose actions are never completed teaches people that post-mortems are theatre.

The PagerDuty template sections are: owner, meeting time, call recording, overview, what happened, contributing factors, resolution, impact (with exact numbers), responders, timeline (with a data link per line), "how did we do" (what went well, what did not), action items, and the messages to send. It also has a status field that moves from draft, to in review, to reviewed, to closed. I use the same skeleton in the example below, with a few changes.

What makes a good action item

Bad: "Be more careful with config." Good: "Make the HTTP client refuse unknown config keys at startup (owner: Dana, due 2026-03-28, ticket PLAT-512)."

Classify actions so that you cover several layers, not only prevention:

• Prevent it from happening again (a guard, a test, a rule).
• Detect it faster (an alert, a dashboard).
• Mitigate it faster (a rollback tool, a kill switch).
• Process (a runbook change, training).

And for each, say how you will know it worked.

A complete fictional post-mortem

Here is a full post-mortem for an invented incident. The company "Shoply", the people, the numbers and the service names are made up for teaching. I tied it to the retry lesson that comes next, and to the SLO numbers from the earlier lessons so the maths agrees. The timeline is the same one used by the script above.

Post-mortem: Checkout failures after the v2.41 release

Summary

On Saturday 14 March 2026, between 14:09 and 14:46 UTC, about 38% of checkout requests failed for 37 minutes. The cause was a combination of a brief slowdown in the payments service and a release of checkout-api (v2.41) in which the payments client was accidentally changed to retry five times with no delay. The retries multiplied the load on the payments service and turned a two-minute slowdown into a 37-minute incident. We rolled back to v2.40, and error rates fell within minutes. The incident used about 32% of the 30-day error budget for checkout.

Impact

(All numbers in this example are invented for teaching. The error budget line comes from the real arithmetic in incident-times.mjs: 0.38 x 37 = 14.1 minutes.)

What happened

checkout-api calls payments-svc to authorise a card. Before the release, the client used up to 3 attempts, with exponential backoff and random jitter, as described in the platform guide.

Release v2.41 (deployed at 14:02 UTC) included a refactor that moved HTTP client settings into a shared config file. The key backoffMs was renamed backoff_ms. The code that reads the config ignored unknown keys and used a default of 0 ms when the setting was missing. At the same time the default number of attempts was changed from 3 to 5 in the same refactor. The result: after any failed payment call the client retried immediately, up to five times.

At about 14:08 payments-svc had a short slowdown (about 2 minutes) while a database replica was promoted. Normally, the old client would have backed off and the slowdown would have passed with no customer impact. With v2.41, every slow or failed call became up to five calls in under a second. Our traffic to payments-svc rose to roughly four to five times normal. Because payments-svc was now overloaded, its calls kept failing even after the database was healthy, which caused more retries. The system stayed in a bad stable state until we removed the retries by rolling back.

Contributing factors

1. Silent config default. An unknown or missing key fell back to 0 ms without an error or a log line.
2. No retry budget or backoff test. Our tests checked "retries happen", but not "retries wait". No test covered the shape of retry traffic.
3. Retries at one level, without a limit. Nothing capped total retry volume, such as a circuit breaker, a retry budget or load shedding on payments-svc.
4. Brief dependency slowdown. The trigger. It was ordinary and expected, and the system should have absorbed it.
5. Release timing. The release went to 100% of pods at once, 7 minutes before the slowdown. A staged rollout (5%, then 25%, then 100%) would have limited the blast radius and made the connection to the release more obvious.
6. Detection delay. The page came 8 minutes after the first failures. Errors grew as the retry load built up, and the fast-burn alert needs both a one-hour and a five-minute window to be above the line and has a 2 minute for clause.
7. Runbook gap. The checkout runbook said "check dependencies" but not "check what was just deployed", and had no rollback command.

Resolution

At 14:24 the on-call responder declared a SEV-2 and Omar took the IC role. At 14:41 the team compared the shape of outgoing traffic before and after the deploy and noticed the five-fold jump in calls per checkout, and found the new retry setting. At 14:46 the rollback to v2.40 completed. Error rates fell within a few minutes. The incident was closed at 14:58 after the error ratio had been under the SLO line for 10 minutes.

Timeline (UTC, 2026-03-14)

Derived numbers (from the script): detection 8 minutes, acknowledgement 2 minutes, mitigation 37 minutes, resolution 49 minutes.

What went well

• The page arrived and was acknowledged within 2 minutes, and a clear IC took over within 7 minutes of the page.
• Roles were used. Priya's notes made this timeline easy to build.
• The customer liaison posted honest updates with the next update time.
• Rollback worked on the first try. Nobody tried to hot-fix forward.
• The ledger check confirmed no customer was charged for a failed order.

What did not go well

• It took 32 minutes from the first failure to find the cause. We looked at payments-svc first. The checkout team's own deploy was not at the top of the checklist.
• Nobody suspected "our retries are making it worse" until the traffic multiplier was seen on a graph. We had no panel for "outgoing calls per incoming request".
• The status page was updated 24 minutes after impact started.

Where we got lucky

• It happened on a Saturday morning with lower traffic than a weekday peak. At weekday volume the payments service would likely have crashed rather than just slowed down.
• A payment provider call that fails is not captured, so there was no double charging.

Action items

Eight or nine items is already a lot. Items 1 to 3 and 6 are the ones the team agreed matter most.

Lessons learned

1. A retry is a bet that the dependency will be healthier in a moment. Without a delay, a limit and a jitter, the bet becomes a weapon pointed at your own dependency.
2. Silent defaults are dangerous. A system that starts with a wrong setting and says nothing is worse than a system that refuses to start.
3. When something is slow, check what we changed and what we are sending, before blaming the dependency.

How we kept this blameless

We name people only by role in the narrative. The config rename was reviewed and approved; the review process did not show the effect of an ignored key, and nothing in the tooling could have shown it. The question we asked was "what made this look safe?" The answer was a quiet default and an incomplete test, so the actions fix those, not the person.

That is a full post-mortem. It's about two pages. Notice what it contains: numbers, links to evidence, several contributing factors, luck stated openly, action items with owners, dates and a way to check them, and no person is blamed.

Common mistakes with incidents and post-mortems

• No owner. Without a named owner, the post-mortem is never written. Assign it at the end of the incident.
• A post-mortem only for "big" incidents. Near-misses teach cheaply. Write short versions for them.
• A blame hunt in disguise. Phrases like "should have known" or "failed to" point at a person. Rewrite them as conditions.
• Action items that are wishes. "Improve monitoring" is not an action. "Add alert X by date Y, owner Z" is.
• Too many actions. Choose the few that really reduce risk and track them to the end.
• Never reading old post-mortems. Keep them searchable, and look through them when you design a new system or plan a game day.
• Skipping the "what went well". Leaving out the good things means you may later "fix" them away.
• Forgetting to update runbooks and alerts. The document is only useful if the system changes afterward.

Try it

1. Write a one-paragraph definition of "incident" and a three-level severity table for a system you work on. Base each level on user impact.
2. Pretend a bad deploy breaks your app right now. Write the first ten minutes as a numbered list: who acknowledges, who is IC, which channel, what is the first mitigation, and how will you know it worked?
3. Rewrite these blaming sentences as blameless ones: (a) "Sam pushed a bad migration." (b) "The on-call engineer ignored the alert." (c) "QA missed this bug."
4. Take the timeline in the example post-mortem. If the alert had paged at 14:12 and the rollback was done at 14:30, recompute time to detect, time to mitigate and the equivalent full-outage minutes (error ratio 38%).
5. Pick one real post-mortem from a public list and find: the trigger, two contributing factors, one thing that went well, and one action you would add.

Answers

1. Any answer that is short, written down, and based on user impact is good. Example: SEV-1 (most users cannot use the core feature), SEV-2 (a large group or an important feature is hurt), SEV-3 (small impact with a workaround).
2. A good answer: acknowledge the page, declare an incident and open one channel, name an IC (or yourself), check what changed (the deploy), roll back, confirm with the error ratio, post an update. It is the same shape as the first-hour list.
3. (a) "A migration shipped that locked a table, and our review and staging setup did not show the lock time." (b) "The alert fired during an unrelated noisy period and the on-call had learned to expect false pages, so it took longer to treat as real." (c) "Our tests did not cover this path, and we have no check that new features include tests for failure cases."
4. Time to detect = 14:12 minus 14:09 = 3 minutes. Time to mitigate = 14:30 minus 14:09 = 21 minutes. Equivalent full-outage minutes = 0.38 x 21 = 8.0 minutes, which is 8.0 / 43.2 = about 18.5% of the monthly budget (versus 32.5% in the real timeline). Faster detection and rollback saved about 14 percentage points of the budget.
5. Open-ended. A good check is that every contributing factor you list is about a system or a condition, not a person.

Interview questions

Key takeaways

• An incident needs roles (IC, deputy, scribe, experts, liaison), one channel, and regular updates. The IC coordinates and does not usually fix.
• Mitigate first, understand later. A fast, safe rollback beats clever live debugging.
• When you are unsure about severity, treat it as the higher one.
• Make small, reversible changes, one at a time, and confirm recovery with the SLI.
• On-call should be sustainable: primary and secondary, real rotation size, handoffs, rest, and improving one thing every shift.
• Track detection, acknowledgement, mitigation and resolution times, and budget used per incident. Beware of the mean (MTTR).
• A blameless post-mortem asks what in the system made the mistake easy. It lists contributing factors, not one root cause, and has a short list of action items with owners and dates.
• Retries and timeouts appear in many real post-mortems, including our fictional one. The next lesson shows how to get them right.

What's next

Next: Reliability Patterns: Timeouts, Retries, Circuit Breakers and Graceful Degradation. We will rebuild the exact failure from the post-mortem in code: 500 clients retrying a dependency, first with no delay and then with backoff and jitter. Then we write a circuit breaker, a bulkhead and a load shedder, and test them with fault injection. Practise now: take the action items from the fictional post-mortem and pick two you would do first, and say why.