SLIs, SLOs and Error Budgets

Ask a team "is the service reliable?" and you get answers like "mostly", "yes, except on Mondays" or "the CEO says no". None of those can be graphed, argued over or used to decide anything. Ask "did 99.9% of checkout requests succeed this month?" and you get a number, a yes or a no, and a clear next step.

This is lesson 4 of nine. The three lessons before it gave us the raw data: logs, metrics and traces. This lesson turns the data into targets. The next lesson, Alerting and Dashboards That People Trust, uses the targets to decide who gets woken up. Lesson 5 goes deep on burn-rate alert rules, so here I only explain what a burn rate is and why it is a good number. The lesson after that, Incidents, On-Call and Blameless Post-Mortems, uses the budget to rate how bad an incident was.

You need no maths beyond percentages. Every number in this lesson came out of a small script I ran, and I show the scripts.

Words used in this lesson

• An SLI (service level indicator) is a measurement of how well the service is doing, from the user's point of view. It is usually a ratio: good events divided by valid events.
• An SLO (service level objective) is a target for an SLI over a time window, such as "99.9% of requests succeed over 30 days".
• An SLA (service level agreement) is a promise to a customer, usually in a contract, with a penalty such as a refund if you miss it. It is looser than your internal SLO.
• Good event, valid event, bad event. A valid event is one that counts at all. A good event is a valid one that went well. A bad event is a valid one that did not.
• Availability is the share of requests (or time) in which the service works.
• The nines is the habit of writing targets as 99.9% ("three nines"), 99.99% ("four nines") and so on.
• Error budget is how much failure the SLO allows: 100% minus the SLO. For 99.9% it is 0.1%.
• Burn rate is how fast you spend the error budget, compared with the speed that would use it up exactly at the end of the window. Burn rate 1 means "on plan". Burn rate 10 means "ten times too fast".
• SLO window is the period the SLO covers. It can be a rolling window (the last 30 days, always moving) or a calendar window (this month).
• Error budget policy is a short written agreement about what the team does when the budget gets low or runs out.
• Toil (a word from SRE) is repetitive manual work that grows with the service and has no lasting value. We only touch it at the end.

Where the three letters come from

Three terms, three audiences:

Notice the order: you first measure (SLI), then you decide how good is good enough (SLO). The SLA, if you have one, is set looser than the SLO. If your SLO is 99.9% and your contract says 99.5%, you will notice and fix trouble before you owe anybody money. An SLA that is as tight as the SLO means you find out about problems at the same moment you start paying for them.

Not every service has an SLA. Almost every service benefits from an SLO, even a private one. An SLO is a tool for your own team to decide how much risk is fine.

Choosing an SLI: start from the user

The best SLI is a number that goes down when users are unhappy and stays flat when they are not. It sounds obvious. In practice teams often choose what is easy to collect: CPU use, memory, disk, number of pods. These are useful for debugging, but users do not feel CPU. They feel a page that does not load.

A simple recipe is to think in user journeys and pick the right shape of SLI for each:

You do not need all of them. For most web services, two cover most of what users feel: availability and latency. Start there.

Write the SLI as "good divided by valid"

Almost every good SLI has this shape:

SLI = good events / valid events

This has three nice properties. It is always between 0 and 1 (or 0% and 100%), so every SLI looks the same. It scales: ten requests or ten million requests, the same formula. And it makes you think about two questions that matter a lot: what counts as valid? and what counts as good? Most arguments about SLOs are really arguments about those two words.

Let me show how much they matter. I generated 200,000 fake requests for one day of a checkout service. About 2% are health checks from the load balancer, 3% are bots, and the rest are real user traffic. Some user requests fail with a 500 (our fault), some return 404 (the user asked for something that does not exist), a few return 429 (we rate-limited them), and about 1% are slow. The script is sli.mjs:

total requests: 200000naive  availability (all traffic, 5xx bad): 99.9145%valid events: 189632  bad: 233careful availability SLI: 99.8771%

Same day, same traffic, two numbers: 99.9145% and 99.8771%. The naive version counted everything and called only 5xx bad. It was too kind in two ways. The denominator included health checks and bots, which almost never fail. And it ignored the 429 responses, which are failures from the user's point of view even though they are a "client error code", because we chose to turn those users away.

My careful version made these rules, written down so everybody can argue about them:

• Valid means: a real user request. Health checks and bot traffic are excluded. Also excluded: 404 and 403 answers, since the user or the bot asked for something that was never ours to give.
• Bad means: a 5xx response, or a 429 (we refused a real user).

You may disagree with those exact rules. Good. The point is that the rules are visible. If you write "availability" on a dashboard and nobody knows what is in the denominator, the number will be used for a decision and the decision will be wrong.

Latency SLIs: use thresholds, not averages

Lesson 2 showed why averages lie. The same rule applies here. A latency SLI is not "average latency is below 200 ms". It is again a good-divided-by-valid ratio: the share of requests that finished under a threshold.

From the same fake day:

latency SLI  < 100  ms: 62.083%latency SLI  < 200  ms: 91.855%latency SLI  < 300  ms: 97.486%latency SLI  < 500  ms: 98.777%latency SLI  < 1000 ms: 99.539%p50 82 p90 185 p99 639 p99.9 1236 mean 109.8

The mean is 109.8 ms and sounds great. But 2.5% of requests were slower than 300 ms, and p99 is 639 ms. Now compare what the two ways of speaking promise:

• "p99 latency under 300 ms" is a statement about one number from a percentile. It is hard to add up over time, because you cannot average percentiles from different minutes into a correct monthly percentile (lesson 2).
• "97.5% of requests are faster than 300 ms" is a ratio of counts. You can add counts over any period and any number of servers. That is why SLO tooling prefers it, and why a histogram with a bucket boundary exactly at your threshold (0.3 s) is the right metric to have.

Both say almost the same thing, but only the second one adds up correctly.

A subtle point: the thresholds and targets come as a pair. "97% under 300 ms" and "99.9% under 1 second" can both be SLOs for the same endpoint. This is a normal and good pattern: most requests should be quick, nearly all should be not-terrible.

Combining success and speed

A user does not care why a request was bad. A fast error and a slow success both hurt. Some teams therefore define one SLI: a request is good if it succeeded and was fast enough. On the same day:

combined good-event SLI (ok and <300ms): 97.3665%

That is the number of requests that did what the user wanted, in time. It is lower than either part alone, as it must be. One combined SLI is simple to alert on and to explain. Two separate SLIs are easier to debug. Pick whichever your team will actually read, and do not spend a month deciding.

How to produce the SLI in Prometheus

If your service exports the usual http_requests_total{code=...} counter and an http_request_duration_seconds histogram (lesson 2), you can store the SLI as recording rules. Here is mine. The two ratios use the same "good over valid" shape:

groups:  - name: checkout-sli    rules:      - record: job:checkout_requests:rate5m        expr: sum(rate(http_requests_total{job="checkout"}[5m]))      - record: job:checkout_bad_requests:rate5m        expr: sum(rate(http_requests_total{job="checkout", code=~"5.."}[5m]))      - record: job:checkout_availability:ratio_rate5m        expr: |          1 - (            job:checkout_bad_requests:rate5m            /            job:checkout_requests:rate5m          )      - record: job:checkout_latency_good:ratio_rate5m        expr: |          sum(rate(http_request_duration_seconds_bucket{job="checkout", le="0.3"}[5m]))          /          sum(rate(http_request_duration_seconds_count{job="checkout"}[5m]))

I checked the syntax and then tested it with promtool test rules, feeding in made-up series: 990 good and 10 bad requests every minute, and a histogram where 950 of every 1000 requests fall in the le="0.3" bucket.

Checking sli-rules.yml  SUCCESS: 4 rules found  SUCCESS

The test expected an availability of 0.99 and a latency-good ratio of 0.95, and passed. To be sure the test could fail, I changed the expected latency to 0.90, and it printed FAILED with exp: 9E-01 got: 9.5E-01. One real-world detail: in a promtool test, the expected labels of a recorded series have to include the metric name (I wrote labels: 'job:checkout_availability:ratio_rate5m'), otherwise a test that looks right reports a failure. My first try with labels: '{}' failed that way.

Notice that I left out the health-check and 404 filtering, to keep the rules short. In real life you would add label filters (route!="/healthz", code!~"404|403") to match the "valid" definition you wrote down. The slok/sloth project, which I read about, generates this kind of recording rules and the alert rules from a short SLO spec, for several time windows at once. That is useful when you have many services. For one or two, hand-written rules are fine.

From SLI to SLO: how good is good enough?

An SLO is a target on the SLI over a window. The shape is always the same: an SLI, a target, and a time window. For example:

99.9% of valid checkout requests succeed, measured over a rolling 30 days.97%   of valid checkout requests finish in under 300 ms, over a rolling 30 days.

How do you pick the number? The honest answer: by asking what users need and what you can pay for, and then adjusting with data.

• Look at what you actually achieved over the last few months. If you ran at 99.95%, a 99.9% SLO is realistic. A 99.99% SLO would be a leap that needs money and design change.
• Remember the SLO is not a wish. It is a trade-off. Each extra nine costs more, and (see below) it also slows you down, because you can afford less change.
• Your service cannot be more reliable than the things it depends on, unless you design for it (see the dependency section).
• Users are often fine with less than you think. An internal report page does not need the same SLO as a payment API. A phone connection, the user's own Wi-Fi and the browser fail often anyway, so past some point extra reliability on your side is invisible to the user.
• Start with a lower target you can meet and tighten later. It is much easier to raise a target than to explain a missed one.

Rolling or calendar window?

• A rolling window (the last 30 days, recomputed all the time) matches how users remember: "it has been bad lately". It never "resets", so a bad day keeps counting until it falls out of the window 30 days later. This is smooth and hard to game.
• A calendar window (this month) is easy to put in a report, and the budget refreshes on the first day. The weakness: a team that blew the budget on the 3rd has no reason to be careful until the 1st of next month, and a team that is fine on the 29th can ship something risky because "it resets tomorrow".

Neither is wrong. Many teams use a rolling 28 or 30 days for alerting and decision-making, and a calendar month for reporting to others. The important thing is to pick one and write it down.

The nines, converted into minutes

Percentages are hard to feel. 99.9% and 99.99% look almost the same on a slide, but they are very different in practice. So let us translate. The allowed bad share is 1 - SLO, and if we count by time, it times the length of the window. Here is nines.mjs:

SLO       per 30 days   per 90 days   per year90        3.00 d        9.00 d        36.50 d99        7.20 h        21.60 h       3.65 d99.5      3.60 h        10.80 h       1.83 d99.9      43.20 min     2.16 h        8.76 h99.95     21.60 min     1.08 h        4.38 h99.99     4.32 min      12.96 min     52.56 min99.999    25.9 s        1.30 min      5.26 min

Learn two rows by heart. 99.9% is about 43 minutes a month. 99.99% is about 4 minutes a month. Each extra nine divides the allowed downtime by ten.

Now think about what four minutes means. A human gets paged, wakes up, finds a laptop, opens a dashboard and starts to look. That alone takes more than four minutes. So with a 99.99% target and a human in the loop, one incident can use the entire month. To meet four nines you need automatic failover and automatic rollback, not heroic on-call. At five nines (26 seconds a month) humans are out of the picture altogether.

That is why the cost of each nine is not small. Consider what you must buy as you go up:

Not every request or user is equal, either. "Time-based" availability counts minutes of downtime. "Request-based" availability counts requests. They behave differently. My policy.mjs script has the same 10-minute total outage at two times of day:

10-minute total outage:at peak                traffic 3500 req/min  budget used (request-based): 40.51%   (time-based: 10/43.2 min = 23.15%)at the quietest time   traffic 500 req/min  budget used (request-based): 5.79%   (time-based: 10/43.2 min = 23.15%)

A time-based view says both outages cost the same: 10 of the 43.2 allowed minutes (23%). A request-based view says the peak outage hurt seven times more, since seven times more users were affected. Request-based is usually fairer, because it counts what users felt. It also means the budget "in minutes" is only an estimate. The budget in requests is the real thing. (In my fake traffic the daily average is about 2,000 requests a minute, so a peak minute costs more than an "average" minute. That is why the 40.5% is more than 23%.)

Your dependencies set a ceiling

A request that needs three services in a row works only if all three work. For independent parts, you multiply their availabilities:

Serial chain: a request needs ALL parts to work3 x 99.9%          99.7003% downtime/30d: 2.16 h5 x 99.9%          99.5010% downtime/30d: 3.59 h10 x 99.9%         99.0045% downtime/30d: 7.17 h99.99 + 3 x 99.9   99.6903% downtime/30d: 2.23 hParallel (two independent replicas, either is enough)2 x 99% -> 99.9900%2 x 99.9% -> 99.9999%

Three dependencies at 99.9% each give you 99.7%. Ten give you 99.0%. And the last serial line shows the main trick for SLO setting: one 99.99% part does not rescue three 99.9% parts. The weakest links dominate. In the other direction, putting two independent copies in parallel turns two 99% parts into 99.99%. That is the whole reason for redundancy.

Two cautions on those numbers. First, they assume the parts fail independently. Two replicas that run the same code on the same database fail together, so the parallel result is too optimistic. Second, real systems also degrade gracefully (lesson 7): if the recommendations service is down, you can still show the page without recommendations, so that dependency is not really "in series". A good way to raise your effective availability is to make dependencies soft instead of hard.

Error budgets: the allowed amount of failure

Now the central idea. If the SLO is 99.9%, then 0.1% of requests may fail and you are still within target. That 0.1% is the error budget.

error budget = 1 - SLO

The word "budget" is the point. A budget is something you are supposed to spend. A team that finishes the month with 100% of its budget unspent was too careful: it could have shipped more, tried more, taken more risks. A team that spent 300% was too reckless. The target is to spend it wisely.

Here is a month in my invented checkout service: 30 days, request counts that rise and fall through the day (between 500 and 3,500 requests a minute), a background error rate of 0.02%, and four incidents. The script is budget.mjs:

requests in window: 86,400,000budget (allowed bad requests): 86,400bad requests actually: 53,185SLI for the month: 99.9384%budget spent by each cause (share of the full-month budget)bad deploy            3.0%slow leak             1.9%dependency down      35.9%cache stampede        0.8%background errors    20.0%

The SLO was 99.9%, the result was 99.9384%, so we met it and have 38% of the budget left. Read the list of costs. Five different causes, and one of them, a 10-minute total outage of a dependency, ate 36% of the whole month's budget by itself. The "bad deploy" (2% errors for 40 minutes) cost only 3%. A short, total failure is far more expensive than a long, small one. And look at the last line: 20% of the budget went on background errors, the steady drip of 0.02% nobody ever paged for. Incidents are not the only budget-spenders. A small constant leak is a real cost, and the budget makes it visible.

Here is the same month day by day. "Budget used of month" is the share of the full month's budget that had been spent by then:

day  SLI-so-far   budget-used-of-month  1    99.9800%          0.7%  3    99.9800%          2.0%  4    99.9578%          5.6%  5    99.9622%          6.3% 10    99.9711%          9.6% 11    99.9719%         10.3% 12    99.9678%         12.9% 19    99.9723%         17.5% 20    99.9727%         18.2% 21    99.9217%         54.8% 26    99.9329%         58.1% 27    99.9338%         59.6% 30    99.9384%         61.6%

Between day 20 and day 21 the budget jumped from 18% to 55% in one day: that is the dependency outage. This is the shape of most reliability stories: long flat stretches, then a step. A line like this on a dashboard tells everybody, in one second, how much room the team has.

Burn rate: how fast are we spending?

The budget is a stock, like money in a bank. The burn rate is the speed of spending. If errors run at exactly the allowed rate (0.1% for a 99.9% SLO), the budget lasts the whole window. That is a burn rate of 1. If errors run at 1%, we burn ten times faster. Burn rate is simply:

burn rate = current error ratio / error budget

burn.mjs shows the table for a 99.9% SLO over 30 days:

error ratio   burn rate   whole budget gone in  0.05%         0.5x    60.0 days  0.10%         1.0x    30.0 days  0.20%         2.0x    15.0 days  1.00%        10.0x    3.0 days  5.00%        50.0x    14.4 hours 50.00%       500.0x    1.4 hours100.00%      1000.0x    43.2 minutes

The last row is the number from the nines table: at a total outage, the monthly budget (43.2 minutes) is gone in 43.2 minutes. A 1% error rate sounds small, a clean "99% success", but at a 99.9% SLO it burns ten times too fast and the month's budget is gone in three days.

The burn rate has a big advantage over the raw error rate: it does not depend on the SLO. "5% errors" is serious for a 99.9% service and a disaster for a 99.99% service. "Burn rate 50" means the same level of urgency in both cases. That is why lesson 5 builds alerts on burn rate: one rule, any service. I will not repeat the alert rules here.

What the budget is for

This is the part most beginners miss. Error budgets are not mainly a way to blame anybody. They are a decision tool for the argument that every company has:

• Product says: "ship faster, we are losing to competitors."
• Operations says: "slow down, every release is a risk."

Without a number, the loudest voice wins. With an error budget, both sides agree on the rules in advance:

• If there is budget left, ship. Go ahead with the risky migration, the experiment, the dependency upgrade. Failure is expected inside the budget. That is what it is for.
• If the budget is nearly spent or gone, slow down. Put effort into reliability: fix the top cause of failures, add the missing test, improve rollback. New features wait.

An agreement like this is called an error budget policy. Mine is short on purpose. A real policy is a page, agreed with the product owner before anything goes wrong, so that nobody has to invent it at 2 a.m.:

Budget policy for checkout (30-day rolling window, SLO 99.9%)- Under 50% used:      normal work.- 50% to 80% used:     risky changes need a second reviewer and a rollback plan.- 80% to 100% used:    only low-risk changes. Fix the top cause of bad requests.- 100% used or more:   feature freeze. Only reliability work and security fixes ship                       until the SLI is back above target.- A single incident that uses over 20% of the budget gets a written post-mortem.

I tested the idea on a rougher version of the same month, with an extra 90-minute incident on day 23 where 35% of requests failed. policy.mjs applies the policy as time goes by:

day 21 09:13  50% used: review risky changesday 24 13:16  80% used: only low-risk changesday 24 13:31  100% used: feature freeze, reliability work onlyend of month: used 183.2% of budget, SLI 99.8168%

Look at the dates. The outage on day 21 crossed 50%. The extra incident then took the team from 80% to 100% in fifteen minutes. A budget policy does not prevent that, because nothing can. What it provides is the rule for what happens next: the freeze starts at 13:31 on day 24 and nobody needs a meeting to decide. The SLO was missed (99.82% against 99.9%), and the missed SLO is not a disaster by itself. It is a signal that says "stop adding risk and fix reliability now".

Note how this solves the political problem. The freeze is not "operations being difficult". It is something that the product owner agreed to when the policy was written. And in the other direction: when the budget is healthy, operations cannot say "no" to a reasonable launch just because change is scary.

Budgets also say something about over-achievement. If a service has run at 99.99% for a year against a 99.9% SLO, it has been spending a tenth of its budget. Either the SLO is too low, or the team has been too careful and could have shipped faster. A few teams even do planned small outages to keep dependants honest, so that nobody quietly relies on a level of reliability that was never promised. You do not need to do that, but the thought is useful: reliability above what users need is waste.

Low traffic makes budgets jumpy

The budget is a count of bad requests, and when counts are small, each single failure is large. burn.mjs shows:

Low-traffic service, SLO 99.9%, 30 days    500 req/day -> 15,000 per month, budget = 15.0 bad requests, one failure = 6.7% of the budget   5000 req/day -> 150,000 per month, budget = 150.0 bad requests, one failure = 0.7% of the budget 500000 req/day -> 15,000,000 per month, budget = 15000.0 bad requests, one failure = 0.0% of the budget

With 500 requests a day, a 99.9% SLO allows 15 failures a month. Two bad minutes can eat it. The SLI becomes noisy and alerts become either useless or constant. For small services, the fixes are: use a longer window (a quarter), use a looser SLO, add synthetic probes (a script that sends a test request every minute, so there is always a steady flow of events), or count at a higher level (one SLO for the whole product, not one per tiny endpoint). The opposite problem also exists: a huge service where 0.1% is still 15,000 failed requests, which may be many unhappy real people. Percentages hide absolute numbers, so keep the counts visible too.

Measuring things, and deciding which ones to promise

A few practical rules from running SLOs:

1. Keep the count small. Two to four SLOs per service (usually availability and latency of the most important journeys) are better than 30. A long list nobody reads is the same as no list.
2. Pick critical journeys. "Login", "search", "checkout", "upload". An SLO on "the whole API" can look healthy while checkout is on fire, because the large amount of cheap, easy traffic hides the failures. If one endpoint matters much more than the rest, give it its own SLO.
3. Write down the definition in one place. What is valid, what is good, where it is measured, which window, who owns it. The OpenSLO project exists to put this in a standard YAML format; I could not read its specification this week, so I only mention it as something to look up.
4. Review the SLO now and then. After each quarter ask: did the SLO match user complaints? If users were angry while the SLO was green, the SLI is measuring the wrong thing. If the SLO was red and nobody cared, it may be too strict.
5. Do not set SLOs for things users cannot feel. "CPU under 70%" is a resource alert or a dashboard line, not an SLO.

The first sign that an SLI is wrong is the "watermelon" effect: green on the outside, red on the inside. The dashboard says 99.95% and the support inbox is full. That means you are measuring something other than what users experience, so go back to the SLI.

Errors and toil: the other use of the budget

One more way to use an error budget is to compare the cost of each cause. In the month above, a short dependency outage cost 36%, and the steady background leak cost 20%. That changes what you should fix first. A dashboard of "budget spent by cause" tells you where an engineering week buys the most reliability. This is more honest than fixing whatever was loudest last week. It also connects to lesson 6: each incident can be reported as "this used 36% of the monthly budget" which gives people who were not there a feel for the size without a long story.

SRE teams use the word toil for manual, repetitive work that grows with the service. A common rule of thumb, from the same SRE material, is to keep toil below about half of an engineer's time so that the rest goes to automation and reliability projects. I could not verify the exact number this week, so treat "about half" as a guide only. The link to this lesson: when budget policy forces reliability work, the right target is often the thing that causes toil and incidents at the same time.

Common mistakes

• Measuring what is easy, like CPU, instead of what users feel.
• Not saying what "valid" means. Health checks and bots in the denominator make the SLI look better than the truth. Our two numbers differed by almost four hundredths of a percent, which is more than a third of the whole 99.9% budget.
• Using averages or a single percentile for latency, instead of the share of requests under a threshold.
• Choosing too many nines. Each one costs ten times more and allows ten times less change.
• SLO equals SLA. Keep the internal target stricter than the promise.
• No policy. An SLO with no agreed consequence is a number on a wall.
• Everything is an SLO. Pick a few critical journeys.
• Calendar-month surprises. Teams who think the budget "resets" and do risky things in the last days.
• Never looking at the green side. Unspent budget over many months is a signal that you can move faster.

Try it

1. A service has an SLO of 99.95% over 30 days. How many minutes of total outage does it allow? What is the burn rate when the error ratio is 0.5%, and how long until the budget is gone?
2. You have 1,000,000 valid requests a day. You see 1,800 of them fail in one bad hour. At a 99.9% SLO over 30 days, what share of the monthly budget did that hour use?
3. Your product calls a payment provider (99.95%), your own database (99.99%) and a search service (99.9%) in series for every order. What is the best availability you can promise for orders, ignoring your own code?
4. Define good and valid events for a "photo upload" journey and for a "daily report email" job.
5. Your SLO dashboard is green at 99.97% but support tickets about slow pages doubled. List three things you would check about the SLI.

Answers

1. The budget is 0.05% of 43,200 minutes, which is 21.6 minutes. A 0.5% error ratio is 0.5 / 0.05 = a burn rate of 10. At burn 10, a 30-day budget is gone in 3 days.
2. The monthly budget is 0.1% of 30,000,000 requests, which is 30,000. The bad hour used 1,800, which is 6% of the monthly budget.
3. 0.9995 x 0.9999 x 0.999 = 0.99840 about, so roughly 99.84%. That is less than 99.9%, which is the target of the weakest link, so you cannot honestly promise 99.9% unless you make one of those dependencies soft (for example, a fallback for search).
4. Photo upload: valid = an upload attempt with a file of allowed type and size; good = it was stored, readable, and finished within a time limit such as 10 seconds. Daily report: valid = a scheduled report; good = the email was sent before 08:00 and had the right data (a freshness and correctness SLI).
5. (a) Is the SLI only availability, with no latency target? (b) Is the latency measured as an average or a high threshold that hides slowness, and is the threshold in line with what users call slow? (c) Is it measured close to the user, or only inside the app, and does the denominator include cheap health checks or bots?

Interview questions

Key takeaways

• Pick SLIs from the user's side: availability and latency of the main journeys. Write them as good events over valid events, and define "valid" on paper.
• Latency SLIs are the share of requests under a threshold, not an average, and your histogram needs a bucket at that threshold.
• An SLO is an SLI, a target and a window. Never 100%. Rolling windows are harder to game.
• Learn the nines: 99.9% is about 43 minutes a month, 99.99% about 4. Each nine is ten times less room and a lot more cost.
• Dependencies in series multiply: three at 99.9% make 99.7%. Redundancy in parallel and soft dependencies raise it.
• The error budget is 1 minus the SLO, and it is meant to be spent on change. Count it in requests, and show the percentage remaining and the cost per cause.
• Burn rate = error ratio / error budget. It works for any SLO, which is why alerts use it.
• Write an error budget policy with the product owner while it is calm, so a freeze is a rule and not an argument.
• If the dashboard is green and users are unhappy, the SLI is wrong, not the users.

What's next

Next: Alerting and Dashboards That People Trust. We use the budget and the burn rate from this lesson to decide who should be woken up, with real alert rules checked by promtool, and a simulation that shows which alerts fire during a blip, a slow leak and a real outage. Practise now: write one availability SLI and one latency SLI for the most important endpoint in a service you know. For each, write down what is valid, what is good and where you would measure it. Then work out its budget in requests per month.