Alerting and Dashboards That People Trust

Most teams do not have too little alerting. They have too much. A phone buzzes at 3 a.m. for a disk that is 81% full, a pod that restarted once, a CPU spike that went away in two minutes. After a few weeks the people on call stop trusting the phone. Then one night a real outage arrives and nobody moves fast, because the phone has cried wolf for months.

This lesson is about making alerts rare and meaningful. We will build the alert design from the ground up, run real rules through promtool (the Prometheus rule checker and tester), run a small simulation to compare a naive alert with a burn-rate alert, and then talk about dashboards, because a dashboard is what you open the moment the phone rings.

This is lesson 5 of a nine-lesson series. It builds directly on SLIs, SLOs and Error Budgets, where we turned "reliable" into a number. If the words "error budget" or "SLO" are new, read that lesson first. The metrics in the examples come from Metrics and Prometheus. After this, Incidents, On-Call and Blameless Post-Mortems covers what happens after the page is sent.

Words used in this lesson

• An alert is a rule that says "if this condition is true, tell someone". The rule lives in a monitoring system.
• A page is an alert that interrupts a human right now, usually a phone call or push message. A ticket is an alert that can wait until working hours.
• Alert fatigue is what happens when people get so many alerts that they stop reacting to any of them.
• A symptom is what the user feels (slow pages, errors). A cause is the reason inside the system (a full disk, a bad deploy).
• Actionable means a human can do something useful about it right now.
• A runbook is a short written guide: "when this alert fires, check this, then try that".
• Burn rate is how fast you are spending the error budget, compared with the "exactly on plan" speed. Burn rate 1 means you will use up the whole budget exactly at the end of the SLO window.
• Multiwindow alert means the alert needs two time windows to both agree, a long one (is this big enough to matter?) and a short one (is it still happening now?).
• A recording rule in Prometheus saves the result of a query as a new metric, so alert rules stay short and fast.
• Alertmanager is the Prometheus program that receives fired alerts and decides who gets told: it groups, routes, silences and inhibits them.
• Inhibition means "if alert A is firing, do not bother people with alert B, it is just a side effect".
• A dashboard is a screen of charts that shows the health of a system.
• RED and USE are two small checklists for what to chart. RED is Rate, Errors, Duration, for services. USE is Utilisation, Saturation, Errors, for resources like CPU or disks.

Why alerts go wrong

Let us start with a picture of a bad week. This is made-up data, but every number is in a small script I ran (incident-times.mjs), and any on-call person will recognise the shape:

pages last week: 48, led to real action: 3 (6%)

Forty-eight pages and only three needed a human. The other forty-five were noise. In plain words, the on-call person was woken up forty-five times for nothing. Here is what the 48 looked like:

The cure is not "better humans". The cure is a rule you can write on a sticky note:

The Prometheus project's own alerting guidance says the same in different words. In my own summary: keep alerting simple, alert on symptoms that users feel, have good consoles (dashboards) to find the cause, and avoid pages where there is nothing to do. It also says to aim for as few alerts as possible and to leave some slack so a small blip does not page anyone.

Symptoms versus causes

A cause alert says "the database CPU is at 90%". A symptom alert says "checkout requests are failing 3% of the time". Which one should wake you?

The symptom. Here is why:

• CPU at 90% might be perfectly fine. The database might be doing exactly what it should.
• Checkout failing is always a problem. You do not need to guess.
• If the database CPU does cause failures, the symptom alert catches it too. You lose nothing.
• There are hundreds of possible causes (CPU, disk, memory, network, a bad deploy, a full connection pool). You cannot write an alert for every one, and you will always miss the next new one. There are only a few symptoms: "errors", "slow", "wrong answer", "data is late".

So the rule is: page on symptoms, use causes for dashboards and tickets. Causes still matter, but as help for the person who is already looking at a symptom.

Also notice what the Prometheus page says about where to measure. For a serving system, alert on errors and latency as high up the stack as possible, which means as close to the user as possible. If a lower service is slow but the final page the user sees is fast (because of a cache, say), there is no user pain, so there is no page. Page on latency at one point in the stack, not at every layer, or one slow database will page the database team, the API team, the web team and the mobile team all at once.

Page, ticket or nothing

Not everything belongs in the same channel. A simple three-level scheme works for most teams:

If you cannot say which row an alert belongs to, it is probably not ready to exist.

A trick for cleaning up: look at the last month of pages. For each one write down "what did the human do?". If the answer is "nothing", "waited", "restarted something that a machine could restart" or "closed it", then delete the alert, or lower it to a ticket, or automate the fix (a restart that is always the answer should be a health check that restarts the process, not a human).

The anatomy of a good alert

A Prometheus alert rule is small. Here is one, simplified:

- alert: CheckoutErrorBudgetFastBurn        # a short, clear name  expr: job:http_errors:ratio_rate1h > 0.0144 # the condition  for: 2m                                    # how long it must stay true  labels:    severity: page                           # routing: who gets told, and how  annotations:    summary: "Checkout is burning its error budget 14x too fast"    runbook_url: "https://wiki.example.com/runbooks/checkout-errors"

Each part has a job:

• Name. Say what is wrong in the user's world: CheckoutErrorBudgetFastBurn, not Alert42. A common community habit is CamelCase names.
• expr. The condition. If it returns any rows, the alert is "pending" or "firing".
• for. The condition must stay true for this long before the alert fires. It filters out blips. But note the trade-off: for: 10m means you always find out at least ten minutes late. Use small values, and let the burn-rate windows do the main smoothing.
• labels. Used for routing. severity: page goes to the pager, severity: ticket goes to a queue.
• annotations. Human text. A summary and, most important, a runbook link.

A checklist for every new alert

Before you merge a new alert rule, ask these seven questions. If you cannot answer one, stop.

1. What user-visible problem does this mean? (If you can't say, it is a cause, not a symptom.)
2. Is it urgent? Would a delay of 8 hours matter? If not, make it a ticket.
3. What will the on-call person do first? If "nothing", delete it.
4. Where is the runbook? It should be linked from the alert itself.
5. Which dashboard shows the details? Link that too.
6. How often will it fire? If it will fire weekly and need no action, it is noise.
7. Has it been tested? Not in production. With a unit test (we do this below).

Runbooks

A runbook is a short page that says: what this alert means, how to check if it is real, the first three things to try, who to call if those fail. It does not need to be beautiful. It needs to be findable at 3 a.m. by someone half asleep who has never seen this alert before.

A short runbook skeleton:

Alert: CheckoutErrorBudgetFastBurnMeaning: more than 1.44% of checkout requests fail (the 30-day budget is going 14x too fast).Check first: the "Checkout overview" dashboard. Is it all regions or one?Was there a deploy in the last hour? -> roll back (command below).Is a dependency down (payments, inventory)? -> open its dashboard; see "Dependency down" section.Still stuck after 15 minutes? Page the checkout secondary on-call, and start an incident (lesson 6).

Two rules for runbooks: put them next to the alert (a URL in annotations), and update them after every incident. An out-of-date runbook is worse than none, because it sends a tired person in the wrong direction. Lesson 6 makes this part of the post-mortem.

Why plain thresholds fail

The first alert most people write is a threshold: "error ratio over 1% for 5 minutes". It looks reasonable. Let us see how it behaves. I wrote a simulation of one day of traffic (1,000 requests per minute, sim.mjs, with a fixed random seed so you get the same output). Three bad things happen in that day, one after the other:

• Minute 100. A tiny blip. 3% errors for 2 minutes. Nobody cares.
• Minute 300 to 900. A slow leak. 0.7% errors for 10 hours. Small each minute, but it goes on and on.
• Minute 1100 to 1130. A real outage. 20% errors for 30 minutes.
• At all other times a normal background of 0.05% errors.

We have a 99.9% SLO over 30 days, so the error budget is 0.1%. Let me first show the budget maths, which I printed with burn.mjs:

SLO 0.999 error budget 0.0010000000000000009 window hours 720budget in minutes of total outage: 43.2error ratio   0.05%  burn    0.5x  budget gone in  1440.0 herror ratio   0.10%  burn    1.0x  budget gone in   720.0 herror ratio   0.50%  burn    5.0x  budget gone in   144.0 herror ratio   1.00%  burn   10.0x  budget gone in    72.0 herror ratio   1.44%  burn   14.4x  budget gone in    50.0 herror ratio   5.00%  burn   50.0x  budget gone in    14.4 herror ratio 100.00%  burn 1000.0x  budget gone in     0.7 h

(The ugly 0.0010000000000000009 is normal floating point noise from 1 - 0.999. It is not a bug.)

Read the table slowly. Burn rate = error ratio divided by the error budget. At 0.1% errors you burn at 1x: you use the full budget in exactly 30 days, which is the plan. At 1% errors the burn is 10x and the budget lasts only 3 days. At 100% errors (total outage) the burn is 1000x and the whole month's budget is gone in 43 minutes.

Now the simulation. Here is what a naive alert does, compared with the two burn-rate alerts we will build:

naive: 5m error ratio > 1%                     min 101-104, min 537-537, min 717-717, min 765-766, min 1100-1133fast burn: 1h>14.4x AND 5m>14.4x (page)        min 1104-1133slow burn: 6h>6x AND 30m>6x (page)             min 611-746, min 759-792, min 808-906, min 1105-1158errors today: 10630 of 1440000 requests; that is 24.6% of the 30-day budget2 min blip used 0.2% of budget, leak used 9.5%, outage used 14.0%

What does this tell us?

1. The naive alert paged for the blip (minutes 101 to 104). The blip used only 0.2% of the monthly budget. It should not wake anyone.
2. The naive alert paged at random during the leak (minutes 537, 717 and 765). The leak sits at 0.7%, close to the 1% line, so random noise makes the 5-minute ratio jump above 1% now and then. Those are meaningless one-minute pages, and then the alert goes quiet again, even though the leak is still happening. Worse, the leak used 9.5% of the budget, nearly as much as the real outage, and the naive alert gave no steady signal about it.
3. The fast-burn alert ignored the blip and the leak and fired only for the real outage (minute 1104, four minutes after it began).
4. The slow-burn alert caught the leak. It fired at minute 611, about five hours after the leak began, because it needs the 6-hour window to fill up with enough bad requests. That is a real delay, and the price of not being noisy. It also flapped off and on around the line (at 7x the leak is only slightly above its 6x threshold), which is normal for a leak that sits near a threshold.

The burn-rate idea in one sentence

Choose how much of your budget you are willing to lose before you wake somebody, and how quickly. Then compute the burn rate that does exactly that:

burn rate needed = (fraction of budget consumed) x (SLO window in hours) / (alert window in hours)

I used that formula in burn.mjs for three common choices (a 30-day window is 720 hours):

threshold table (fraction of 30d budget consumed within the alert window):page: 2% in 1h       burn >= 14.40  => error ratio >= 1.440%page: 5% in 6h       burn >= 6.00  => error ratio >= 0.600%ticket: 10% in 3d    burn >= 1.00  => error ratio >= 0.100%

Read it as plain sentences:

• "Page me if we have used 2% of the month's budget in the last 1 hour." That needs a burn of 14.4. For a 99.9% SLO, that is 1.44% of requests failing.
• "Page me if we have used 5% in the last 6 hours." Burn of 6, which is 0.6% failing.
• "Open a ticket if we have used 10% in the last 3 days." Burn of 1, which is the plan speed, so a ticket says "you are on course to spend everything, look at it this week".

These are the numbers I remember from the Google SRE Workbook chapter on alerting on SLOs (see my note above that I could not read it this time). You are free to choose other percentages. What matters is that you choose the percentage of budget on purpose, and the maths turns it into a rule.

Why two windows

If you use only a long window (1 hour), the alert has a bad habit: it keeps firing for up to an hour after the problem is fixed, because the bad minutes are still inside the window. If you use only a short window (5 minutes), it is noisy again.

So use both, joined with and:

• The long window (1h) says "this is big enough to matter".
• The short window (5m, which is about 1/12 of the long one) says "and it is still happening right now".

The alert fires only when both are above the line, and it clears as soon as the short window recovers. We will prove that clear behaviour with a unit test below.

The real rule file

Writing rules directly against raw counters is slow and repetitive, so we first save the error ratio for each window as a recording rule, then write the alerts against those. This is my file slo-alerts.yml. The metric http_requests_total is the counter from the metrics lesson, with a code label.

groups:  - name: checkout-slo-recording    rules:      - record: job:http_errors:ratio_rate5m        expr: sum(rate(http_requests_total{job="checkout",code=~"5.."}[5m])) / sum(rate(http_requests_total{job="checkout"}[5m]))      - record: job:http_errors:ratio_rate30m        expr: sum(rate(http_requests_total{job="checkout",code=~"5.."}[30m])) / sum(rate(http_requests_total{job="checkout"}[30m]))      - record: job:http_errors:ratio_rate1h        expr: sum(rate(http_requests_total{job="checkout",code=~"5.."}[1h])) / sum(rate(http_requests_total{job="checkout"}[1h]))      - record: job:http_errors:ratio_rate6h        expr: sum(rate(http_requests_total{job="checkout",code=~"5.."}[6h])) / sum(rate(http_requests_total{job="checkout"}[6h]))  - name: checkout-slo-alerts    rules:      # SLO 99.9% => budget 0.001. 14.4 x 0.001 = 0.0144      - alert: CheckoutErrorBudgetFastBurn        expr: job:http_errors:ratio_rate1h > (14.4 * 0.001) and job:http_errors:ratio_rate5m > (14.4 * 0.001)        for: 2m        labels:          severity: page        annotations:          summary: "Checkout is burning its 30-day error budget 14x too fast"          runbook_url: "https://wiki.example.com/runbooks/checkout-errors"      # 6 x 0.001 = 0.006      - alert: CheckoutErrorBudgetSlowBurn        expr: job:http_errors:ratio_rate6h > (6 * 0.001) and job:http_errors:ratio_rate30m > (6 * 0.001)        for: 15m        labels:          severity: page        annotations:          summary: "Checkout is burning its error budget 6x too fast"          runbook_url: "https://wiki.example.com/runbooks/checkout-errors"

Notes on the rules:

• The name pattern job:http_errors:ratio_rate1h is the community habit for recording rules: level:metric:operations. It is a habit, not a law.
• The rate() over a window gives requests per second averaged over that window. Dividing errors per second by requests per second gives the error ratio. (We sum first, so we divide totals and not per-pod ratios, which would give the wrong answer when pods have different traffic.)
• Putting 0.001 in the expression makes the budget visible. When someone changes the SLO to 99.95%, the person editing the file sees exactly which numbers to change.

First, check that the file is valid:

$ promtool check rules slo-alerts.ymlChecking slo-alerts.yml  SUCCESS: 6 rules found

And to prove that check really catches mistakes, here is a deliberately broken rule (a missing bracket):

$ promtool check rules bad.ymlChecking bad.yml  FAILED:bad.yml: 5:11: group "x", rule 1, "Bad": could not parse expression: 1:18: parse error: unclosed left parenthesis

Unit tests for alerts

check only proves the file is valid. It does not prove the alert does what you think. For that, promtool has test rules. You give it fake metric values over time, and say "at minute X, this alert should (or should not) be firing". It is like a unit test for your pager.

The syntax 0+5400x120 means "start at 0, add 5400 each step, repeat 120 times", which is a counter that goes up by 5,400 per minute (90 requests per second). I wrote two scenarios in slo-alerts.test.yml (the second one has two checks):

rule_files:  - slo-alerts.ymlevaluation_interval: 1mtests:  # Scenario 2: a real outage (20% errors for 40 minutes) MUST page, then CLEAR.  - interval: 1m    input_series:      - series: 'http_requests_total{job="checkout",code="200"}'        values: '0+4800x120'          # 80 ok req/s      - series: 'http_requests_total{job="checkout",code="500"}'        values: '0+0x10 0+1200x40 48000+0x70'    alert_rule_test:      - eval_time: 45m            # outage is on: must page        alertname: CheckoutErrorBudgetFastBurn        exp_alerts:          - exp_labels:              severity: page            exp_annotations:              summary: "Checkout is burning its 30-day error budget 14x too fast"              runbook_url: "https://wiki.example.com/runbooks/checkout-errors"      - eval_time: 62m            # outage ended at minute 50: the 5m window is clean, so it must clear        alertname: CheckoutErrorBudgetFastBurn        exp_alerts: []  # Scenario 3: steady 1% errors for 7 hours (burn rate 10).  # Too slow for the fast alert, right for the slow alert.  - interval: 1m    input_series:      - series: 'http_requests_total{job="checkout",code="200"}'        values: '0+5940x500'      - series: 'http_requests_total{job="checkout",code="500"}'        values: '0+60x500'    alert_rule_test:      - eval_time: 400m        alertname: CheckoutErrorBudgetFastBurn        exp_alerts: []      - eval_time: 400m        alertname: CheckoutErrorBudgetSlowBurn        exp_alerts:          - exp_labels:              severity: page            exp_annotations:              summary: "Checkout is burning its error budget 6x too fast"              runbook_url: "https://wiki.example.com/runbooks/checkout-errors"

(The comment numbers start at 2 because my first scenario, a short blip, turned out to be useless and I deleted it. More on that below.) The result:

$ promtool test rules slo-alerts.test.yml  SUCCESS

A test that always passes is worthless, so I also broke the rules on purpose to see if the tests notice. This is called a mutation test, and it is the only way to know a test has teeth.

Mutation A: remove the short 5-minute window (keep only the 1-hour window). The test for "it must clear after the outage" fails, exactly as the theory predicts:

FAILED:    alertname: CheckoutErrorBudgetFastBurn, time: 1h2m,         exp:[],         got:[            0:              Labels:{alertname="CheckoutErrorBudgetFastBurn", severity="page"}              Annotations:{runbook_url="https://wiki.example.com/runbooks/checkout-errors", summary="Checkout is burning its 30-day error budget 14x too fast"}            ]

Twelve minutes after the outage ended, the one-window alert was still paging. That is the "page keeps ringing after the fix" problem, shown with real output.

Mutation B: lower the fast threshold from 14.4x to 5x. The 1%-for-7-hours scenario now wrongly triggers the fast alert:

FAILED:    alertname: CheckoutErrorBudgetFastBurn, time: 6h40m,         exp:[],         got:[ ... the fast-burn alert is firing ... ]

(I shortened the second output with .... The real output has the same labels and annotations as the first.)

Alertmanager: what happens after the alert fires

Prometheus decides whether an alert is firing. It does not send phone calls. It hands firing alerts to Alertmanager, which decides who hears about them and how often. Four ideas matter:

1. Routing. A tree of rules matches labels (severity="page") and chooses a receiver (pager, Slack, email).
2. Grouping. Ten alerts about the same problem become one message, grouped by labels such as alertname and service. This stops "50 pods are failing" from creating 50 pages.
3. Inhibition. If a bigger alert is firing, mute its smaller side effects. If the whole checkout service is down, do not also page for the slow-burn alert of the same service.
4. Silences. A human can mute an alert for a set time ("we know, we are fixing it" or "planned maintenance").

From the Alertmanager configuration file docs, the three timers and their default values are:

Here is my test config, alertmanager.yml. The URL and the key are fake on purpose.

route:  receiver: team-slack  group_by: [alertname, service]  group_wait: 30s  group_interval: 5m  repeat_interval: 4h  routes:    - matchers: ['severity="page"']      receiver: oncall-pager      repeat_interval: 1hinhibit_rules:  # if the whole service is down, do not also page for its slow-burn alert  - source_matchers: ['alertname="CheckoutErrorBudgetFastBurn"']    target_matchers: ['alertname="CheckoutErrorBudgetSlowBurn"']    equal: [service]receivers:  - name: team-slack    slack_configs:      - api_url: https://hooks.slack.example.invalid/T000/B000/XXXX        channel: "#checkout-alerts"  - name: oncall-pager    pagerduty_configs:      - routing_key: not-a-real-key

And the real checks. First that the file is valid, then which receiver each kind of alert would reach:

$ amtool check-config alertmanager.ymlChecking 'alertmanager.yml'  SUCCESSFound: - global config - route - 1 inhibit rules - 2 receivers - 0 templates$ amtool config routes show --config.file=alertmanager.ymlRouting tree:.└── default-route  receiver: team-slack    └── {severity="page"}  receiver: oncall-pager$ amtool config routes test --config.file=alertmanager.yml severity=page alertname=CheckoutErrorBudgetFastBurnoncall-pager$ amtool config routes test --config.file=alertmanager.yml severity=ticket alertname=DiskFillingteam-slack

A page goes to the pager. A ticket-level alert goes to Slack. This was only a routing check. I did not run a live Alertmanager and send a message, because that would need a real Slack or PagerDuty endpoint. So I cannot show you a delivered notification, and the inhibition rule was only checked for valid syntax, not exercised.

More traps that cause bad pages

Low-traffic services

A burn rate is a ratio. If a service gets 3 requests per hour, one failure is a 33% error ratio and the alert screams. For low-traffic services, ratios jump wildly. Options: use a longer window, add a minimum request count to the condition, or use synthetic probes (a fake user that tests the service every minute so there is steady traffic). In an interview, saying "burn-rate alerts need enough traffic to make the ratio meaningful" is a strong answer.

The "for" clause is not a free filter

for: 15m means a real problem is detected at least 15 minutes late. For a page about a 50%-failing checkout that is a long time. Keep for short on fast alerts (a minute or two) and let the windows do the work.

Missing data

If your metrics stop arriving (a broken scrape, a crashed exporter), rate() returns nothing, and an alert with ratio > threshold evaluates to nothing, which means no alert. A silent monitoring system looks exactly like a healthy system. Two protections:

• Alert on the scrape itself: up == 0 for the important jobs.
• Run a "dead man's switch" (watchdog): an alert that is always firing. A receiver outside your system expects to hear it all the time and pages if it stops. The Prometheus guidance calls this idea meta-monitoring: make sure monitoring itself works, ideally with an end-to-end test from Prometheus through Alertmanager to the person.

Alerting on the average

Never alert on average latency. The average hides the slow tail. One user in a hundred waiting 20 seconds does not move an average much, yet that user is gone. Use a percentile (p95 or p99) or, better, the SLO form: "fraction of requests slower than 300 ms". The metrics lesson showed why. See Metrics and Prometheus.

Dashboards that people trust

An alert tells you that something is wrong. A dashboard tells you where to look. A bad dashboard has 60 charts, every metric the system exposes, in random order. Nobody can read it at 3 a.m. A good one has a purpose and a path.

One dashboard, one question

Decide the question first:

Link them in a chain: overview, then component, then deep dive. The on-call person should be able to go from the alert to the cause in three clicks.

What goes on the overview

Start from the golden signals you met in lesson 0. Two helpful checklists:

• RED for anything that serves requests: Rate (requests per second), Errors (failing fraction), Duration (latency, as percentiles).
• USE for resources like CPU, disk, a connection pool: Utilisation (how busy), Saturation (how much work is waiting), Errors.

A sketch of a service overview (this is a text drawing of a layout, not a screenshot of a real tool):

+---------------------------------------------------------------+| Checkout        [last 6h v]   deploys: ^ v2.41 at 14:02       |+----------------+-------------------+--------------------------+| SLO status     | Error budget left | Burn rate (1h / 6h)      || 99.93% (OK)    | 71% of 30 days    | 0.8x   /   0.9x          |+----------------+-------------------+--------------------------+| Requests/sec (by route)            | Error ratio (5xx / all)  ||  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  |  ____________/\____      |+------------------------------------+--------------------------+| Latency p50 / p95 / p99            | Saturation: queue depth, ||  ----  ----  ----                  | pool usage, CPU          |+------------------------------------+--------------------------+| Dependencies: payments  inventory  database  (each: rate/err/p99)|+---------------------------------------------------------------+

Notice the order. Top row: the answer ("healthy or not") in big numbers. Below it, the supporting charts. At the bottom, dependencies. This is the same order the on-call person asks questions in.

Rules that make dashboards trustworthy

• Show deploys and config changes as markers on the charts. Most incidents follow a change. If you can see the line "v2.41 deployed" next to the spike, you save ten minutes.
• Use percentiles, not averages, for latency. Show p50, p95 and p99 together.
• Put units and thresholds on the chart. "ms", "req/s". Draw the SLO line.
• Keep the same time range and the same colour for the same thing everywhere. If red means "errors" on one chart and "CPU" on another, people misread.
• Prefer a few charts that matter over many that exist. A new chart should replace or earn its place.
• Make the dashboard load fast. A heavy dashboard that takes 30 seconds to render is not used during an incident. Recording rules help here too.
• Keep dashboards in version control ("dashboards as code"), reviewed like code, so changes are visible and nobody "fixes" a chart by accident.
• Do not alert from a dashboard. A chart that "someone should be watching" is not monitoring. Alerts are rules. Dashboards are for looking when you already have a reason.

Beware of "everything is green"

A very common failure: all charts look fine, but users are hurt. This often means you measure from the wrong side. If your errors chart counts only responses that reached the server, a load balancer that rejects requests earlier is invisible. Measure as close to the user as practical (the load balancer or a synthetic probe), and check that a deliberately broken request shows up on the chart. If you have never broken it on purpose, you do not know it works.

Try it

1. Take a service you know. Write its SLO as one sentence: "99.9% of checkout requests succeed over 30 days".
2. Compute its error budget in minutes of total outage for 30 days. (Hint: 30 x 24 x 60 x budget fraction.)
3. Using the formula burn = fraction x window_hours / alert_hours, find the burn rate that means "2% of the budget gone in 1 hour" for a 28-day window. (A 28-day window is 672 hours.)
4. A service gets 12 requests per minute. Your fast-burn alert fires when the error ratio is above 1.44%. How many failed requests in one minute is that? Does it make sense to alert like this?
5. Look at your current alerts. For each, write "what does the on-call person do?". Delete or downgrade the ones with no answer.
6. If you have Go installed, build promtool the way I did, write a test with one scenario of your own, then break the rule and check that the test fails.

Answers

1. Example: 99.9% over 30 days. Anything similar is fine as long as it says what is measured and for how long.
2. 30 x 24 x 60 x 0.001 = 43.2 minutes. (That was the first line of my burn.mjs output.)
3. 0.02 x 672 / 1 = 13.44. Compare with 14.4 for a 30-day window. The shorter window gives a slightly smaller number because 2% of a shorter month is a smaller amount of time.
4. 1.44% of 12 requests is 0.17, so even one failed request in a minute (8.3%) is above the line. That alert would fire from a single failure. For such a low-traffic service use a longer window, a minimum traffic condition, or synthetic probes.
5. There is no single answer. Typical deletions: disk at 80%, single pod restart, CPU high on one node.
6. The mutation exercise is the point. A test that cannot fail does not protect you.

Interview questions

Key takeaways

• A page must be urgent, actionable and about user pain. Everything else is a ticket, a dashboard panel, or nothing.
• Alert on symptoms (errors and latency users see). Use causes on dashboards, except for a few "this will hurt soon" cases.
• Burn rate = error ratio / error budget. It tells you how fast the budget is going.
• Choose how much budget you accept losing and how fast, then compute the threshold: fraction x window_hours / alert_hours. For a 30-day window, 2% in 1 hour is 14.4x, 5% in 6 hours is 6x, 10% in 3 days is 1x.
• Use two windows: a long one for "big enough" and a short one for "still happening", so the alert clears when the problem is fixed.
• Test your alert rules with promtool test rules, and break them on purpose to see the test fail. My first test was too weak.
• Alertmanager groups, routes, inhibits and silences. Know the defaults: 30s, 5m, 4h.
• Beware low traffic, long for: delays, missing data, and averages. Add a dead man's switch.
• A dashboard answers one question and leads you from "is it bad" to "where" in a few clicks. It shows deploys, uses percentiles and is kept in version control.

What's next

Next: Incidents, On-Call and Blameless Post-Mortems. The page has fired and a real person is awake. Who does what in the first hour? How do you talk to customers? And how do you write the document afterwards so the team learns something instead of blaming someone? We will write a full fictional post-mortem together. Practise now: run promtool check rules and a test on one alert of your own before you read on.