Monitoring, Alerting & Blameless Postmortems

Part 1: Monitoring and dashboards

Monitoring means you keep collecting and watching signals (metrics, logs and traces, post 46) to know how healthy your system is. A dashboard is one screen of graphs that shows these signals.

Useful dashboards, from the top down (start with the user, end with the machines):

1. User-journey / SLO dashboard: are users OK? SLIs, error budgets and burn rates for login, search, checkout and so on (post 47).
2. Service overview: for each service, the golden signals (latency as the median and p99, traffic, errors, saturation), plus deploy markers. A deploy marker is a line on the graph that shows when a new version went out.
3. Dependency dashboards: databases, caches, queues and third-party APIs. For a queue, lag means how far behind the readers are. Depth means how many messages are waiting.
4. Infrastructure: nodes, containers, CPU, memory, disks and networks (the USE method).

Design dashboards so an on-call engineer can go from "users are affected" to "it's this service" to "it's this dependency" in a few clicks.

Other monitoring types:

• Synthetic monitoring: scripted "robot users" that log in, search and check out every minute from several places. They find problems even when real traffic is low, like at night.
• Real User Monitoring (RUM): measures what real users' browsers and apps experience. It collects page load times, JavaScript errors and crash rates, split by device, country and network.
• Heartbeat / "dead man's switch" monitoring: you get an alert when an expected signal stops. For example, a nightly backup job did not report success (post 45). Here, silence is the warning.

Part 2: Alerting that people trust

An alert that wakes someone up is called a page. Every page should pass three tests:

1. Urgent: it needs action now, not tomorrow morning.
2. Actionable: there's something a human can actually do.
3. User-impacting (or about to be): it reflects real or imminent pain for users.

If an alert fails any test, it should be a ticket (a task handled during work hours), a dashboard item, or it should be deleted.

Alert on symptoms, investigate causes. A symptom is what users feel (errors, slowness). A cause is the reason inside the system (high CPU, a full disk).

High CPU is only a problem if it hurts users. Put causes on dashboards for investigation, and page on symptoms, ideally SLO burn rates (post 47).

Some cause-based alerts are still worth having, especially predictive ones. A predictive alert looks ahead, for example "disk will be full in 4 hours at the current rate" or "certificate expires in 7 days". These should usually be tickets, not 3 AM pages, unless the deadline is very close.

Every alert needs a runbook. A runbook is a short written guide that tells the on-call person what to do. It explains what the alert means, how to check the impact, the common causes, and the steps to reduce the harm (mitigation). Mitigation steps include rollback, failover, scale up and a kill switch. These are explained in Part 3. With a runbook, anyone on call can respond, not only the expert.

Fighting alert fatigue. Alert fatigue means people get so many alerts that they stop paying attention to them. Here is how to fight it:

• Review alerts regularly: "Which alerts fired? Were they useful? Did anyone act?" Delete or fix noisy ones.
• Group and de-duplicate: one incident shouldn't produce 200 separate pages. Tools like Prometheus Alertmanager or PagerDuty group related alerts. (PagerDuty is a service that sends alerts to the on-call person's phone.)
• Use sensible durations: alert only when the problem lasts "for 5 minutes", not on a single data point.
• Route to the right team, based on service ownership.
• Track pages per on-call shift. A common goal is no more than a couple of pages per shift, so each one gets full attention.

On-call practices:

• Rotations (weekly is common) with a primary and a secondary engineer.
• Escalation policies: these are rules for who gets paged next. For example, if the primary does not acknowledge (reply to) the page within 5 minutes, page the secondary, then the manager.
• Handoffs between shifts: the outgoing person tells the next person about ongoing issues and recent changes.
• A sustainable load: compensate on-call time, and make fixing noisy alerts part of the job.

Part 3: Incident response

An incident is any unplanned event that hurts (or threatens) users or the business. A clear process keeps people calm and organised.

The lifecycle:

Severity levels (SEV) say how serious an incident is, so people know how fast to act. For example:

• SEV1: major outage. Many users cannot use a core feature (like payments). Everyone helps, and you tell the public.
• SEV2: significant degradation or a partial outage.
• SEV3: minor impact, with a workaround available.

Clear roles help in larger incidents. They come from the way emergency services run their incidents:

• Incident Commander (IC): leads the response, makes decisions and keeps the big picture. The IC does not debug (look for the bug).
• Operations / subject-matter experts: investigate and apply fixes.
• Communications lead: updates the status page, customer support and leadership at regular intervals.
• Scribe: writes a timeline of what was seen, tried and decided. This is very useful for the postmortem.

Mitigate first, investigate later. The first goal is to stop user pain, even before you know the root cause. Common quick mitigations:

• roll back the latest deploy. This means going back to the previous version. It is the most common fix, because most incidents follow a change,
• fail over to another zone or region. This means switching traffic to a healthy copy of the system (post 40),
• scale up (add more servers), or shed load (refuse some requests on purpose so the rest still work) (post 44),
• flip a feature flag / kill switch. A feature flag is a setting that turns a feature on or off without a new release. A kill switch is a flag for turning a feature off fast (post 44),
• block abusive traffic (post 43).

Root-cause analysis can happen once users are safe.

Part 4: Blameless postmortems

After a serious incident, write a postmortem. A postmortem is a written review of what happened, why, and what to change. It is also called an incident review or retrospective. The goal is learning, not punishment.

Why blameless? Almost every incident involves a human action. Someone deployed, someone ran a command, or someone missed an alert. But if people are punished for mistakes:

• they hide mistakes and near-misses,
• the real system problems (missing safeguards, confusing tools, noisy alerts, time pressure) never get fixed,
• the same incident happens again with a different person.

A blameless postmortem assumes people acted reasonably with the information and tools they had, and asks how the system made the mistake easy and the impact large.

A postmortem usually has a timeline like this:

Good postmortem habits:

• Look for contributing factors, in the plural. Complex failures rarely have one "root cause" (the single main reason). Safety engineers often use the "Swiss cheese" model. Each layer of defence is a slice of cheese with holes. An incident happens when the holes in several slices line up.
• "5 whys" means asking "why?" again and again. It helps you dig deeper. Do not stop at "human error". Keep asking why the system allowed it.
• Include what went well, so you keep doing it.
• Action items must be specific, owned and tracked until done. A postmortem whose actions nobody tracks is just a story.
• Share widely, internally and sometimes publicly. Other teams learn too.
• Review near-misses as well. They are free lessons.