System Design, Step by Step

Part 8 · Observability

SLI, SLO, SLA & Error Budgets

Product wants to ship daily; operations wants fewer changes; nobody agrees what “reliable enough” means. SLIs, SLOs and SLAs turn it into one number, and error budgets turn that number into a simple rule for when to move fast and when to slow down.

Lesson 4710 min readintermediate#sre#reliability

On this page

  1. The Problem
  2. The Core Idea
  3. How It Works
  4. Trade-offs
  5. In the Real World
  6. Interview Questions
  7. Key Takeaways
  8. Further Reading

The Problem

The product team wants to release new features every day. The operations team wants fewer changes, because most outages start with a change. Every week they argue:

  • "The site is unreliable, stop releasing!"
  • "It's fine! Users aren't complaining!"

Nobody can win, because nobody has agreed on what "reliable enough" means. A service is reliable when it works correctly each time users need it.

Meanwhile, the monitoring dashboard (a screen of graphs) has 400 graphs, and it is not clear which ones really matter to users.

SLIs, SLOs and error budgets end this argument. They turn "reliable enough" into a number everyone agrees on. They also give a simple rule for when to move fast and when to slow down.

The Core Idea

SLI — indicator
what you measure — good events ÷ valid events, e.g. % of checkouts that succeed
SLO — objective
your internal target — 99.95% over a rolling 28 days — alert and act here
SLA — agreement
your promise, with money attached — 99.9% — looser than the SLO
Error budget
100% − SLO — the failure you are allowed to "spend" while you release changes
Three terms that build on each other — plus the budget they create. · layers

Think about a train service.

  • SLI (Service Level Indicator) is what you measure: "the percentage of trains arriving within 5 minutes of schedule".
  • SLO (Service Level Objective) is your target: "95% of trains arrive within 5 minutes, measured each month".
  • SLA (Service Level Agreement) is your promise to customers, with consequences: "if fewer than 90% arrive on time this month, season-ticket holders get a 10% refund".
  • The error budget is the allowed failure. With a 95% target, 5% of trains may be late. Say you still have plenty of "late budget" left this month. Then you can do track work that might cause delays. If you have used it all, the track work waits until next month.

Notice two things:

  1. The SLO is not 100%. No real system is perfect. Chasing perfection costs a huge amount of money and slows everything down.
  2. The SLA is looser than the SLO. This way you notice problems and fix them before you owe customers money.

How It Works

SLI: measure what users experience

An SLI is a number that measures one part of your service, as users experience it. It is usually a ratio of good events to valid events. A "valid" event is one that should count. For example, a request that fails only because the user sent bad input does not count. The valid events are the bottom number (the denominator) of the ratio.

Text
SLI = good events / valid events × 100%

Common SLI types:

SLI"Good event" meansExample
AvailabilityRequest succeeded (not a 5xx, which is an HTTP status code from 500 to 599 that means the server failed)99.95% of checkout requests succeeded
LatencyRequest was faster than a threshold99% of search requests < 300 ms
QualityResponse was complete (not degraded)99.9% of product pages showed prices (not a fallback)
FreshnessData was updated recently99% of dashboard data < 1 minute old
CorrectnessOutput was right99.999% of payments recorded with the correct amount
DurabilityData was not lost99.999999999% (eleven nines) of stored objects retained
Throughput / pipelineJob completed in time99% of nightly reports ready by 7 AM

Tips for good SLIs:

  • Measure as close to the user as possible. Logs from the load balancer (a server that spreads requests across many servers) or from the gateway are better than internal service metrics. Even closer to the user are real-user monitoring (RUM), which measures what real browsers and apps experience, and synthetic probes, which are scripted fake users that call your service on a schedule.
  • Use percentiles for latency, never averages (post 8). A percentile such as p99 means that 99% of requests were faster than that value. Latency means how long a request takes.
  • Leave out invalid events (like 400 Bad Request, which is caused by bad client input) from the ratio.
  • Focus on user journeys: "log in", "search", "add to cart", "check out", "play video". Don't try to write SLIs for every internal component.

SLO: the target

An SLO is a target value for an SLI over a period of time. So it combines an SLI, a target and a time window:

Text
"99.9% of checkout requests will succeed, measured over a rolling 28-day window.""99% of search requests will complete in under 300 ms, over 28 days."

Choosing targets:

  • Start from what users need and what the business can afford. Do not start from "what we get today" or "as many nines as possible". (A "nine" is one digit 9 in a number like 99.9%.)
  • Different journeys get different targets: checkout maybe 99.95%, recommendations 99.5%.
  • Remember your dependencies. A dependency is another service you rely on. You cannot be more available than the services you depend on (post 9).
  • Start achievable, then tighten. An SLO you miss every month gets ignored. An SLO you always beat by a large margin may be too loose, or it may mean you spend too much on reliability.

Rolling vs calendar windows:

  • Rolling (the last 28 or 30 days): the window moves forward every moment, so it always shows recent reliability. This is good for operations.
  • Calendar (per month or quarter): the window is fixed, like "March". It matches business reports and SLAs. The budget "resets" at the start of each period.

SLA: the contract

An SLA is a business agreement with customers, with consequences if it is missed. The usual consequence is service credits (free usage or money back) or refunds.

  • It's written by business and legal teams, informed by engineering.
  • It's looser than the internal SLO. For example, SLO 99.95% and SLA 99.9%.
  • Not every service needs an SLA, but every important user-facing service benefits from an SLO.
  1. 100%
    never the target
  2. Internal SLO
    99.95% — page and fix here
  3. External SLA
    99.9% — credits owed here
The SLO sits above the SLA, so you act long before you owe customers anything. · flow

Error budgets: turning SLOs into decisions

Error budget = 100% − SLO. It is the amount of failure you are allowed. Think of it as money you can spend on risky changes.

Example: SLO = 99.9% of requests succeed over 30 days, with 10 million requests expected.

0.1%
error budget
100% − 99.9%
10,000
failed requests allowed
this month
~43 min
full outage equivalent
0.1% of 30 days
A 99.9% SLO over 30 days with 10 million requests. · numbers

How teams use the budget:

  • Budget remaining → move fast. Release features, run experiments and do risky migrations (moving data or systems to a new setup). Some failures are expected and acceptable.
  • Budget running low → be careful. Slow down rollouts (releasing a change to users) and add more testing.
  • Budget exhausted → an error budget policy kicks in. For example: freeze non-critical launches (stop releasing them), focus engineers on reliability work, and require extra review for changes, until reliability recovers.

The result: product and reliability teams now share one goal. Nobody argues about feelings. They look at the budget. Spending the budget on new ideas is a good thing. An unused budget may mean you are too careful.

Write an error budget policy in advance, agreed by product and engineering leadership, so decisions in a crisis are not a fight between teams.

Burn rate: how fast you're spending the budget

Burn rate tells you how quickly you are using up the error budget. It compares your speed with the steady speed that would use exactly 100% of the budget by the end of the window.

1×
steady
whole budget used in exactly 30 days
14.4×
1.44% errors vs 0.1% allowed
budget gone in ~2 days — page now
2% / hour
at 14.4×
of the monthly budget
Burn rate — how fast you're spending the budget compared with an even pace. · numbers

At a burn rate of 1, you'd use the whole budget in exactly 30 days. At 14.4, you'd use it in about 2 days, and you burn 2% of the monthly budget every hour.

Alerting on SLOs: multi-window, multi-burn-rate

Do not alert on "CPU > 80%" or "one error happened". Alert when you are burning budget fast enough to matter. Google's SRE Workbook recommends combinations like these for a 30-day window. A "window" here is how far back you look when you measure the burn rate:

SeverityBurn rateLong windowShort windowBudget used if it continues
Page (wake someone)14.4×1 hour5 minutes2% in 1 hour
Page6×6 hours30 minutes5% in 6 hours
Ticket (fix during work hours)1×3 days6 hours10% in 3 days
  • The long window confirms it is a real problem that lasts, not a short blip.
  • The short window makes sure the problem is still happening now. This way the alert stops soon after things recover.
  • The result: fewer, more meaningful alerts, each tied directly to user impact (more in post 48).

A simple SLO document

Text
Service:        Checkout APIUser journey:   Customer places an orderSLI:            % of POST /checkout requests returning non-5xx,                measured at the load balancerSLO:            99.95% over rolling 28 daysLatency SLI:    % of POST /checkout requests completing in < 800 msLatency SLO:    99% over rolling 28 daysError budget:   0.05% of requests (~20 minutes of full outage per 28 days)Alerting:       Page at 14.4× burn (1h/5m) and 6× (6h/30m); ticket at 1× (3d/6h)Policy:         If budget exhausted: freeze non-critical deploys to checkout,                prioritise reliability work until SLO is met for 7 daysOwner:          Payments & Checkout teamReview:         Quarterly
  1. 1
    Plenty of budget left
    release features, run experiments, do the risky migration
  2. 2
    Budget running low
    slow rollouts, more canary releases (first try a change on a few users) and review
  3. 3
    Budget exhausted
    freeze non-critical launches; reliability work first until the SLO holds for 7 days
An error budget policy, agreed before it's needed. · flow

Common mistakes

  • Too many SLOs. Five meaningful ones beat fifty ignored ones.
  • SLOs nobody acts on. If missing the SLO changes nothing, it's just a graph. Agree on the policy.
  • Measuring the wrong thing, like server CPU instead of user success.
  • Targets of 100%, which leave no budget, so every change becomes impossible or every failure becomes a crisis.
  • Averages for latency. Use percentile-based SLIs.
  • Ignoring dependencies when setting targets.

Trade-offs

  • Stricter SLOs: happier users, but much higher cost (more redundancy, slower change processes) and a smaller budget for innovation.
  • Looser SLOs: faster development and lower cost, but a risk of users noticing and leaving.
  • Rolling windows: a responsive, operational view. Calendar windows: simpler business reports, but the budget jumps suddenly at each reset.
  • SLO-based alerting: fewer, more relevant pages, but it takes effort to set up, and it's less useful for very low-traffic services (a few errors cause big percentage swings).

In the Real World

Google SRE. Google introduced the SLO and error-budget approach in its Site Reliability Engineering book (2016) and The Site Reliability Workbook (2018). Its key idea, that 100% is the wrong reliability target, changed how many companies think about reliability. Error budgets let product teams launch freely while budget remains. When the budget is spent, the SRE team can say "wait".

Cloud provider SLAs. Major cloud providers publish SLAs (often 99.9%–99.99%, depending on the service and set-up). If they miss one, you get service credits. Inside the company, they aim for stricter SLOs, so SLA breaches are rare.

Status pages. Many companies (GitHub, Slack, Atlassian and others) publish public status pages that show incidents and uptime per component. They make reliability visible and often reflect the SLIs the companies track internally.

Error budget freezes. Many engineering organisations adopting SRE practices use deploy freezes when a service's error budget is exhausted. This is a clear example of a policy turning a number into an action. Some companies also use budgets to justify more risk: "we're well within budget, let's do the migration this week."

Interview Questions

Q1Explain SLI, SLO and SLA with an example.

SLI: the share of checkout requests that succeed, measured at the load balancer. SLO: 99.95% of them succeed over a rolling 28 days — an internal target. SLA: a contractual promise of 99.9% monthly, with service credits if missed. The SLA is looser so you react before owing money.

Q2What is an error budget, and how is it used?

100% minus the SLO — the amount of failure you're allowed. While budget remains, teams ship and take risks; when it's exhausted, an agreed policy kicks in (freeze non-critical launches, prioritise reliability). It aligns product and reliability around one number.

Q3Why is 100% the wrong reliability target?

It's unachievable, users can't tell the difference beyond a point (their own networks are less reliable), and chasing it makes every change risky and very expensive, leaving no room to ship.

Q4What is burn rate, and why alert on it?

The observed error rate divided by the rate the SLO allows. A burn rate of 14.4 over an hour spends 2% of a 30-day budget in that hour. Alerting on burn rate pages people only when user-facing reliability is really at risk, instead of on noisy resource thresholds.

Q5Why use two windows in SLO alerts?

The long window (say one hour) confirms a sustained problem rather than a blip; the short window (five minutes) confirms it's still happening, so the alert clears soon after recovery. Together they give fast, low-noise pages.

Q6What makes a good SLI?

It reflects what users experience on a key journey, is measured as close to the user as possible (edge logs, RUM, synthetic probes), uses percentile thresholds for latency, and excludes invalid events such as client errors.

Key Takeaways

  • SLI = what you measure (good events ÷ valid events), based on user journeys. SLO = your target over a time window. SLA = a contract with consequences, looser than the SLO.
  • 100% is the wrong target. Choose SLOs from user needs and business cost, and give different journeys different targets.
  • Error budget = 100% − SLO. Spend it on innovation, and follow an agreed policy (like freezing launches) when it runs out.
  • Burn rate shows how fast you're spending the budget. Use multi-window, multi-burn-rate alerts (for example 14.4× over 1h/5m) instead of noisy threshold alerts.
  • Keep a few meaningful SLOs, measured close to the user, with clear owners and regular reviews.

Further Reading

  • Site Reliability Engineering by Google (chapter "Service Level Objectives", and chapter "Embracing Risk" on error budgets)
  • The Site Reliability Workbook by Google (chapters "Implementing SLOs" and "Alerting on SLOs")
  • Implementing Service Level Objectives by Alex Hidalgo
  • The Google Cloud documentation on SLO monitoring concepts
  • The OpenSLO specification project on GitHub

Published by Jainik · Last updated · How these lessons are made

Keep learning

Part of the study guide

Related series