Observability and SRE Interview Questions With Model Answers

You have now read eight lessons. This last lesson turns them into seventeen interview questions with model answers, grouped by topic, plus a scenario drill, a one-page cheat sheet, and a mock interview you can run with a friend.

How to use this lesson: cover the answer, say yours out loud for one minute, then compare. Do not memorise my words. Memorise the shape of each answer: a one-line definition, one example with a number, and one trap or trade-off. Interviewers for these roles care about trade-offs more than definitions.

This is lesson 8 of 8, the last one. The ideas come from What Observability Is and the Three Pillars, Logging That Helps, Metrics and Prometheus, Distributed Tracing With OpenTelemetry, SLIs, SLOs and Error Budgets, Alerting and Dashboards That People Trust, Incidents, On-Call and Blameless Post-Mortems and Reliability Patterns. Each answer says which lesson to reread if you are shaky.

Words used in this lesson

• Observability is how well you can understand what a system is doing inside from the data it gives out (logs, metrics, traces), including for problems you did not predict.
• Monitoring is watching known signals and alerting when they cross a line. Observability is what lets you ask new questions after the alert.
• Logs, metrics, traces are the three pillars: events with detail, numbers over time, and the path of one request across services.
• Cardinality is the number of different label combinations a metric has. High cardinality makes metrics slow and costly.
• Golden signals are latency, traffic, errors and saturation.
• SLI, SLO, SLA are the measurement, the internal target, and the contract promise.
• Error budget is the amount of failure the SLO allows.
• Burn rate is how fast the budget is being spent compared with the plan. Burn rate 1 is on plan.
• Symptom alert is an alert about what users feel (errors, slowness). A cause alert is about an internal thing (high CPU) that may or may not hurt anyone.
• Alert fatigue is when people get so many pages that they stop trusting or reading them.
• Runbook is a short written guide that goes with an alert: what it means, how to check, what to do.
• Incident commander (IC) is the person who coordinates an incident and usually does not do the repair.
• Mitigation stops the pain (such as a rollback). Resolution is the real fix.
• Blameless post-mortem is a review that looks for what in the system allowed the failure and does not hunt for a person to blame.
• Timeout, retry, backoff, jitter are the basic tools for calling unreliable things.
• Circuit breaker, bulkhead, load shedding stop a failure from spreading.
• Graceful degradation is giving a smaller but useful answer when a part fails.
• Fault injection is breaking things on purpose, in a test, to prove the safety features work.

The numbers worth knowing by heart

Interviewers love to ask for a calculation. These all come from math.mjs (a 30-day month is 43,200 minutes):

SLO 99   % -> budget 1.00%  = 432.0 min per 30 daysSLO 99.9 % -> budget 0.10%  = 43.2 min per 30 daysSLO 99.95% -> budget 0.05%  = 21.6 min per 30 daysSLO 99.99% -> budget 0.01%  = 4.3 min per 30 dayserror ratio 0.10% -> burn rate 1.0, budget gone in 30.0 dayserror ratio 0.50% -> burn rate 5.0, budget gone in 6.0 dayserror ratio 1.44% -> burn rate 14.4, budget gone in 2.1 dayserror ratio 5.00% -> burn rate 50.0, budget gone in 0.6 daysburn rate 14.4 for 1 h uses 2.0 % of a 30-day budgetburn rate 6 for 6 h uses 5.0 % of a 30-day budget3 deps at 99.9% in series: 99.700 %3 deps at 99.9% + own 99.95%: 99.650 %1000 requests, 1% take 8-12 s: average 236 ms, p50 110 ms, p99 8704 ms, p99.9 11744 ms

How to read the last line: if 1 request in 100 takes about 10 seconds, the average (236 ms) still looks fine and the median (110 ms) looks great. Only p99 (8.7 seconds) shows the problem. That is why we never alert on the average.

The formula for burn rate: burn rate = observed error ratio divided by the budget ratio. For a 99.9% SLO the budget ratio is 0.001. A burn of 14.4 means 1.44% of requests failing.

Questions on monitoring and the three pillars

Q1. What is the difference between monitoring and observability? (Must know)

Monitoring watches things I already know to watch: I choose signals like error rate or CPU in advance and alert when they cross a line. It answers "is it broken?". Observability is the quality of the system that lets me answer new questions from its outputs, without shipping new code: "why are only users in one region slow, and only on the mobile app?". In practice they work together. Monitoring tells me something is wrong, and the rich data (logs, metrics, traces, with context like request ids and versions) helps me find out why. The honest caveat is that "observability" is also a fashionable word, so I judge a system by whether I could debug a new problem with the data it has. (Lesson 0.)

Q2. When would you use a log, a metric, or a trace? (Must know)

I use a metric to know that something is wrong and how big: error ratio, p99 latency, requests per second. Metrics are numbers over time, cheap to store and good for alerts and dashboards. I use a trace to learn where time went or where it failed across services for one request. I use a log to learn what exactly happened in one place, with detail: the error message, the input id, the decision taken. A good investigation goes metric, then trace, then log: the dashboard shows p99 for /checkout went up, a trace shows the slow part is the payments call, and the log of that service shows the timeouts. They connect through shared ids: the trace id goes into every log line. (Lessons 0 to 3.)

Q3. What is cardinality and why does it matter for metrics? (Must know)

Cardinality is how many different time series a metric creates, which equals the number of different label value combinations. A counter with labels method (4 values), route (20) and status (10) can create up to 800 series. If I add a user_id label with a million users, I create a million times more series, and the monitoring system runs out of memory or the bill explodes. So labels must be low-cardinality things I really want to slice by: route template (/users/:id, not /users/42), status class, region. Details about one request go into logs and traces, which are made for that. A quick check after adding a label is to look at how big the /metrics output became. (Lesson 2.)

Q4. Why is average latency a bad number to alert on? What do you use? (Must know)

An average hides the slow tail. In my test, with 1 request in 100 taking about 10 seconds, the average was 236 ms and the median 110 ms, both looking healthy, while p99 was 8.7 seconds. Those slow users are real people, and they are often the heavy or valuable users. I use percentiles (p95, p99) from a histogram, or better, the SLO form: "the fraction of requests slower than 300 ms". The extra reason for histograms is that percentiles cannot be averaged across servers, but histogram buckets can be added together. (Lessons 2 and 4.)

Q5. How do you follow one request across ten services? (Senior signal)

With a trace and a correlation id. The first service creates a trace id (or reads one from the incoming request). It then passes the id on in every outgoing call using a standard header, traceparent in the W3C Trace Context format, which OpenTelemetry's HTTP instrumentation does for me. Each service creates spans (a named, timed step) that point to a parent span, so the tree can be rebuilt. I also put the trace id in every log line, so from one slow trace I can find all its logs. For queues and background jobs, there is no HTTP header, so I copy the context into the message and restore it in the consumer. And I need a sampling plan, because I cannot keep every trace: keep all errors and slow ones, and a small share of the rest. (Lesson 3.)

Questions on SLIs, SLOs and error budgets

Q6. What is the difference between an SLI, an SLO and an SLA? (Must know)

An SLI is a measurement from the user's point of view, usually good events divided by valid events, such as "the share of checkout requests that succeeded in under 300 ms". An SLO is the internal target for that number over a window, for example 99.9% over 30 days. An SLA is a promise to a customer in a contract, with a penalty such as a refund. The SLA should be looser than the SLO, so that I get an internal warning before I owe money. If a team sets the SLA equal to the SLO, there is no safety margin. (Lesson 4.)

Q7. What does 99.9% availability mean in minutes? And 99.99%? (Must know)

For a 30-day month of 43,200 minutes, 99.9% allows 0.1% of failure, which is 43.2 minutes. 99.99% allows 4.3 minutes. And 99% allows about 7 hours (432 minutes). Each extra nine cuts the allowed failure by ten, but it usually costs much more than ten times the effort, because it needs redundancy, careful deploys and automatic recovery, since a human cannot even respond within 4 minutes. If the SLO is request-based, the same percent applies to requests and not to minutes, so a short, complete outage in a quiet hour costs less than the same minutes at peak. (Lesson 4.)

Q8. What is an error budget and what is it for? (Must know)

The error budget is 100% minus the SLO: the amount of unreliability the business accepts. For 99.9% over 30 days it is about 43 minutes of full outage, or 0.1% of requests. Its purpose is to turn the argument "ship faster" against "be safer" into a shared number. While there is budget left, we can take risks: deploy often, run experiments. When the budget is nearly gone, the team slows feature work and spends effort on reliability until it recovers. It also tells us that 100% is the wrong target: users cannot tell 99.99% from 99.999% if their own phone is less reliable than that, and every extra nine blocks change. (Lesson 4.)

Q9. What is burn rate, and how do you use it for alerting? (Senior signal)

Burn rate is how fast I am spending the budget compared with the speed that would use it exactly at the end of the window. It equals the observed error ratio divided by the budget ratio. For a 99.9% SLO, 0.1% errors is burn 1, which is on plan; 1.44% errors is burn 14.4, which empties a 30-day budget in about 50 hours; 5% errors is burn 50. For paging I use two or more windows together. For example: page if burn is above 14.4 over both the last hour and the last 5 minutes (that is 2% of the monthly budget in one hour). The short window makes the alert stop firing soon after the problem ends. A second, slower rule uses burn 6 over 6 hours (5% of the budget) and goes to a ticket or a lower-urgency page. This pages me for fast disasters and not for slow, small leaks. I derived those thresholds with the script above, and I would check the SRE workbook before quoting it as the source. (Lessons 4 and 5.)

Q10. Your service hit its SLO this month with 40% of the budget still unused. Is that good? (Trap)

Not necessarily. Unused budget can mean I am too careful. If the budget is always left over, the target may be too high, or we are shipping less often than we could, or we are over-investing in reliability for this service. The budget is there to be spent on useful risk: faster releases, experiments, migrations. A better question is whether the SLO matches what users need, and whether they were happy. Also I would check the SLI itself: a green dashboard with unhappy customers means I measure the wrong thing, for example server-side success while the real problem is slowness or errors in the browser. (Lesson 4.)

Q11. You depend on three services, each with a 99.9% SLO, all in series. What availability can you promise? (Trap)

Less than 99.9%. If all three must work for a request to succeed, availability multiplies: 0.999 x 0.999 x 0.999 is about 99.7%, even if my own code is perfect. With my own service at 99.95%, it is about 99.65%. So my dependencies set a ceiling on my SLO. To do better I must reduce the number of hard dependencies on the critical path, make some of them optional with a fallback (graceful degradation), add redundancy such as a second provider, or promise a lower number honestly. This is a main reason reliability patterns matter: a fallback turns a hard dependency into a soft one. (Lessons 4 and 7.)

Questions on alerting and dashboards

Q12. What makes a good alert? How do you fight alert fatigue? (Must know)

A good page is urgent, actionable and about a symptom: users are hurt now (or will be soon), a human must act, and it comes with a runbook link. If an alert fires and nobody needs to do anything, it should not be a page. I alert on symptoms (error ratio, latency SLO burn) more than on causes (CPU at 80%), because a symptom alert catches many different causes and a cause alert often fires when nobody is hurt. To fight fatigue I count pages per shift and the share that were actionable, delete or downgrade the useless ones, group related alerts so one outage is one page, and send non-urgent signals to a ticket queue or a dashboard. Every alert needs an owner, and I test alert rules, for instance with promtool test rules. A team where only 3 of 48 pages were actionable has a broken alerting system, and the fix is the alerts, not the people. (Lesson 5.)

Q13. What would you put on a service dashboard? (Senior signal)

I start from the four golden signals: latency (as percentiles, split into good and bad requests), traffic, errors, saturation. For a service people call, RED is the short form: rate, errors, duration. For resources like CPU, disks and connection pools, USE: utilisation, saturation, errors. I put the SLO and the remaining error budget at the top, then the golden signals, then the dependencies (latency and errors of each downstream call, breaker state, retries per request), then the resources. I mark deploys on the graphs, since most incidents follow a change. And I keep one dashboard for one purpose with a few clear panels, because a wall of 60 graphs helps nobody at 3 a.m. (Lessons 0 and 5.)

Questions on incidents and post-mortems

Q14. Walk me through the first ten minutes of a serious incident. (Must know)

I acknowledge the page so it is clear a human is on it. I check the dashboard to confirm it is real. I declare an incident with a severity and open one channel. I say who the incident commander is; if I am alone, that is me. Then I ask what changed recently: a deploy, config, a feature flag, traffic, a dependency. I choose the quickest safe mitigation (a rollback, a flag off, shifting traffic) and give it to a named person. Someone writes the timeline. I post a first status message with the time of the next update. The key idea is mitigate first, understand later: stopping the harm comes before finding the cause, and I keep the evidence to study afterwards. I confirm recovery with the SLI, not with a feeling, and I wait a short agreed time before closing. (Lesson 6.)

Q15. What is a blameless post-mortem? Does that mean no accountability? (Must know)

No. Blameless means I assume people did the reasonable thing with the information and tools they had, and I ask what in the system let a single mistake cause harm. I replace "who did this?" with "what made this look like a reasonable action at that moment?". The answer is usually a missing guard, a confusing name or an out-of-date runbook, and those are things I can fix. Accountability stays, but it means finishing the action items, not punishing a person. If I punish, people hide mistakes and near-misses, and the weakness remains. A good document has numbers for the impact, a timeline with links to evidence, several contributing factors instead of one "root cause", a short list of actions with owners, dates and a way to check each, and a section on what went well and where we were lucky. (Lesson 6.)

Questions on reliability patterns

Q16. A dependency is slow. Explain how timeouts, retries and circuit breakers work together, and what can go wrong. (Must know)

First a timeout on every call, a little above the healthy p99, so I stop waiting and free my resources; a deadline shared by the whole request is even better. Then retries for failures that may pass, but only for idempotent calls (or with an idempotency key), with few attempts, exponential backoff, full jitter, and a retry budget so retries stay a small share of traffic. Then a circuit breaker around the dependency: when failures pass a threshold, it opens and fails calls instantly, then after a wait lets one trial call through (half-open) and closes if it works. What can go wrong is mostly retries. In my experiment, 5 immediate retries per user lifted the load to about 3.9 attempts per user, and the server kept refusing requests for several seconds after the real fault had ended. With backoff, jitter and a budget, the extra load was 1.07 attempts per user and recovery was immediate. Retries also multiply across layers: three layers with three attempts each send 27 calls to the bottom service. So I retry in one layer, and I export attempts per request as a metric. Last, I add a fallback so that an open breaker gives the user something useful. (Lesson 7.)

Q17. How do you prove that your resilience features really work? (Senior signal)

By fault injection, because an untested safety feature is a guess. I make the dependency fail on purpose, in tests or in a planned game day: instant errors, but also hangs, slowness without errors, flapping up and down, and partial failure like 10% of calls. For each experiment I check three things: did the system protect itself (the breaker opened, load stayed bounded, a bulkhead kept other features working), did we notice (an alert fired, a panel showed it), and did it recover by itself when the fault ended. I also watch for false comfort. In my own test the retry storm did not appear at all until I made each rejected request cost some CPU, because with free rejections the server recovered immediately. A test that is too kind proves nothing. For a game day I start small, tell people, and have a clear stop button. (Lesson 7.)

Scenario drill: "checkout is slow"

Interviewers often give a vague problem and watch how you think. Practise with this one. Say your method out loud, in this order. I put the moves in a short list and show the kind of answer that works.

The prompt: "Since 2 p.m., some users say checkout takes 10 seconds. Walk me through it."

1. Confirm and size it. "Is it real and how big? I look at the checkout SLO dashboard: error ratio and p99 latency for the last few hours. Say p99 went from 400 ms to 9 seconds at 14:00, and errors are flat. That is a latency problem, not an error problem. I check whether p50 moved too. If only p99 moved, a small share of requests is slow."
2. Ask what changed. "Deploys, config or flag changes at 14:00, a traffic jump, a dependency incident. The deploy markers on the graph answer that in seconds."
3. Narrow with the golden signals. "Is traffic up? Is saturation high (CPU, connection pool, queue length)? Is it one region, one version, one route?" Splitting by labels is what the metric labels are for.
4. Find where the time goes with a trace. "I open a slow trace from the last hour, filtered to checkout over 5 seconds. The waterfall shows the payment call taking 8.7 seconds. So the time is in the call to the payments service, not in my code."
5. Zoom with logs. "I take the trace id into the logs of the payments client. I see timeouts and retries, 3 attempts each, no backoff."
6. Mitigate first. "If a deploy changed the retry setting, roll back. If payments is slow, reduce retries, open the breaker, or use the fallback. I act, then watch the SLI come back."
7. Communicate. "A status update with the next update time, and one channel."
8. Afterwards. "A blameless post-mortem with contributing factors: why was a 9-second latency not caught by an alert? Do we have a latency SLO burn alert? Do we have a panel for calls per request?"

Notice that the answer uses all of the series in order: SLO, metrics, trace, logs, incident habits, reliability patterns, post-mortem. That is exactly what a strong candidate shows.

A one-page cheat sheet

Common mistakes in these interviews

• Naming tools and nothing else. "We use Grafana and Prometheus" does not show you understand why. Start with the question you want to answer, then name the tool.
• No numbers. Say "99.9% over 30 days is about 43 minutes", not "very few failures".
• Only the happy path. Always say what happens when the thing fails, and how you would know.
• Blaming a person in a story. When asked about an incident, describe system conditions and what you changed. A blame story is a red flag.
• Claiming a perfect system. Say what you did not have, what you would add, and what you chose not to build and why.
• Forgetting cost. Logging everything, tracing everything and alerting on everything have real costs. Mention sampling, retention and cardinality.
• Mixing up SLO and SLA, or MTTR as a goal. The mean hides long incidents, so look at the spread and the budget used per incident.
• Copying my words. If you say my sentences you will freeze when asked a follow-up. Own the idea with your own example from a real system.

Try it: a mock interview

Do this with a friend, or alone with a timer.

1. Pick any five questions above at random. Answer each aloud in 60 to 90 seconds without looking. Record yourself if you can.
2. For each answer, count whether you gave: a definition in one line, one concrete example with a number, and one trade-off or trap. Mark the ones that missed.
3. Take the scenario "the API returns 500 errors for 3% of requests since the last deploy" and walk through it with the eight-step method above. How long does it take you to say "roll back" and why that comes first?
4. Compute without the script: a 99.95% SLO over 30 days. How many minutes of full outage does it allow? A 2% error ratio for 30 minutes uses how many percent of that budget?
5. Describe, in three sentences, an alert you would delete from a system you know, and why.
6. Explain to a beginner, in two sentences each, a timeout, a retry with jitter, and a circuit breaker. If you need a third sentence, simplify.

Answers

1. Compare with the model answers. The good sign is not matching my words but hitting the same three parts: definition, example with a number, trade-off.
2. This is a self-check. Redo the ones that missed a part.
3. Confirm the 3% in the dashboard and when it began, note the deploy at that time, and roll back (or turn off the flag) as the first safe action. Then use a trace and logs on the failing requests to understand the cause, after users are no longer hurt. Rolling back first is the point: mitigation before understanding.
4. 0.05% of 43,200 minutes is 21.6 minutes. A 2% error ratio for 30 minutes is like 0.02 x 30 = 0.6 minutes of full outage, which is 0.6 / 21.6, about 2.8% of the budget. (Check: burn rate is 0.02 / 0.0005 = 40; 40 for half an hour is 40 x 0.5 / 720 = 2.8%.)
5. Open-ended. A good answer names a cause alert that fires when nobody is hurt (for example CPU above 80% for 5 minutes), says it is not actionable and trains people to ignore pages, and replaces it with a symptom alert or a dashboard panel.
6. Example: "A timeout is a limit on how long I wait for an answer, so a hung service cannot hold me forever. A retry with jitter tries again after a short random wait, so many clients do not all come back at the same second. A circuit breaker stops calling a service that keeps failing, so I fail fast and the service can recover."

Key takeaways

• Interviewers want shape, numbers and trade-offs: a one-line definition, an example with a number, and a trap.
• Know the nines in minutes (99.9% is 43.2 minutes per 30 days) and the burn-rate formula (error ratio divided by budget ratio).
• Never alert on averages. Use percentiles or threshold SLIs, and page on symptoms with a runbook.
• In an incident: roles, one channel, mitigate first, confirm with the SLI, write a blameless post-mortem with contributing factors and owned actions.
• Reliability patterns work together: timeout, retry with backoff and jitter and a budget, circuit breaker, bulkhead, load shedding, and a fallback. Retries are the dangerous one.
• Prove it with fault injection: hangs, slowness, flapping and partial failures, not only instant errors.
• Use the whole chain in scenario answers: SLO, metrics, trace, logs, mitigation, communication, post-mortem.

What's next

This is the last lesson of the Observability and SRE series. To keep going:

• Practise the same method on slow-system scenarios in The Debugging Method: Measure, Isolate, Hypothesize, Verify, which uses the same "measure first" habit.
• Build the logging, error handling and structure that make a service observable in the Backend in Depth series.
• Learn the platform side of running services (images, health checks, rollouts, probes) in Docker and Kubernetes From Zero and see where deployment and hosting fit in the DevOps pages.
• Test failure paths on purpose with the Testing series.
• Practise telling your incident stories well in Behavioral Interviews for Developers.
• Or revisit any lesson in this series, starting from What Observability Is and the Three Pillars.

Your practice for today: pick the one question you answered worst in the mock interview, rewrite your answer in three parts (definition, example with a number, trade-off), and say it aloud until it takes under a minute.