Production Incidents: Step-by-Step Guide

Senior interviews love incident scenarios because they test your judgement under pressure. What do you do in the first five minutes? How do you decide to roll back? What changes afterwards? This lesson gives you the playbook. Then it gives eleven realistic scenarios in the Symptom, Diagnose, Fix, Prevent shape.

Words used in this lesson

Incident words

• Incident is an unplanned event that harms or threatens the service. Outage is when the service is down. Degraded means it works, but badly (slow, or some features fail).
• Mitigate means to reduce the harm now, even if the cause is not yet known. Fix (or resolve) means to remove the cause. Difference in one line: mitigation stops the bleeding; the fix heals the wound.
• Rollback is going back to the previous working version. Roll forward (hot-fix) is shipping a new change that fixes the problem. Difference in one line: rollback undoes the change; roll forward adds another one.
• Postmortem is a written review of an incident: what happened, why, and what to change. Runbook is a step-by-step guide for a known problem. On call means you are the person who is expected to respond to alerts outside normal hours.
• Blast radius is how much of the system or how many users a failure can reach.
• Tenant is one customer (or one company) in a system that serves many customers.

Release words

• Canary release is a rollout where the new version first serves a small share of users, so a bug hurts few people. Progressive rollout grows that share in steps. Smoke test is a quick check that the most basic things work.
• Migration is a change to the database structure. It is backward-compatible when the old version of the code still works with it, which makes rollback safe.

Measuring words

• Latency is how long a request takes. p95 is the value that 95 out of 100 requests beat, so it shows the slow end. RED metrics are Rate (requests per second), Errors and Duration.
• Saturation is how full a resource is (CPU, memory, connections, disk). Event-loop lag is how late Node's single event loop runs its tasks, a sign that JavaScript is blocking it.
• Trace is a record of one request as it moves through the services, with the time spent in each step. Synthetic check is a scripted fake user that tests the service on a schedule.

Traffic and failure words

• Rate limiting means refusing requests above a set rate. The server answers 429 (too many requests), and the Retry-After header says when to try again. Load shedding means dropping some work on purpose so the rest can succeed.
• Retry storm is what happens when many clients retry at once after a failure and make it worse. Backoff means waiting longer between each retry. Jitter is a random extra delay, so clients do not all retry at the same moment.
• Circuit breaker is code that stops calling a failing service for a while, so calls fail fast. Bulkhead is a separate pool of resources for each dependency, so one slow dependency cannot use up everything. Difference in one line: a circuit breaker stops calls; a bulkhead limits how much damage they can do.
• Cascading failure is when one failure makes the next part fail, and so on.
• Idempotent means doing the same operation twice has the same result as doing it once. An idempotency key is a unique ID sent with a request so a repeat can be spotted.
• Stale-while-revalidate is a cache rule that lets the cache answer with an old copy at once while it fetches a fresh copy in the background.
• Failover is switching to a standby copy when the main one fails. Primary is the database that accepts writes. A replica is a copy that follows the primary.
• Connection pool is a set of open database connections that the app reuses. A pooler is a middle service that shares a few real connections among many clients.
• Dead-letter queue (DLQ) is a side queue that holds messages that keep failing. A poison message is one that always fails.
• WAF (web application firewall) is a filter that blocks bad web requests. Rotate a secret means to replace it with a new one and stop using the old one. Least privilege means giving each person or program only the access it needs.
• PostgreSQL words: VACUUM is a command that cleans up dead row versions. Autovacuum runs it automatically. ETL (extract, transform, load) is a job that copies and reshapes data. EXPLAIN (ANALYZE, BUFFERS) shows the plan of a query, the real time and the memory pages it read. Query plan is the way the database chooses to run a query. Statistics are the numbers about your data that the database uses to choose the plan.

The incident playbook

1. Detect and declare

• Someone notices the problem (an alert, a customer report, a dashboard). Your first decision is to declare an incident early. It is cheap to downgrade it later, and expensive to wait.

• Assign roles. Even in a small team, say them out loud:

  • an incident commander (coordinates and decides)
  • an investigator (hands on the keyboard)
  • a communicator (updates stakeholders)

  One person must not do all three jobs.

• Open one channel or "war room" (a shared chat or call just for this incident) and keep a running timeline (what we saw, what we tried, and when). The timeline later becomes the postmortem.

2. Assess impact and severity

Ask three questions: Who is affected? How badly? Is it getting worse? Then give the incident a severity level (SEV means severity; SEV1 is the worst), so everyone has the same picture. The names and the times in this table are examples. They differ between companies.

3. Mitigate before you diagnose

To mitigate means to reduce the harm. The goal is to restore service, not to understand the cause yet. Try these in order, from fastest and safest:

1. Roll back the last deploy (or switch a feature flag off). A feature flag is a switch that turns a feature on or off without a deploy. If something changed shortly before the symptoms began, this is the most likely fix. First check that the release did not include a database migration that the old code cannot work with. If it did, a rollback can fail or lose data, so roll forward instead.
2. Scale or shed load. Add instances, turn on rate limiting, switch off expensive features, or serve cached or static fallbacks.
3. Fail over (switch to a standby copy, such as a replica or another region) if the infrastructure is the cause.
4. Block or isolate the problem: an abusive client, a bad tenant (customer), a poisoned queue message.
5. Only if none of this helps, debug forward and hot-fix (ship a quick fix to the live system).

4. Investigate with a method

Use the loop from Lesson 00: measure, isolate, hypothesize (make a guess that you can test), verify. Start with the question "what changed?" Look at these things:

• deploys
• config and feature-flag changes
• infrastructure changes
• traffic patterns
• an expiring dependency or certificate (a certificate is the file that proves a server's identity for HTTPS)
• a cron job (a task that runs on a schedule)
• data growth

Match the time the symptom began against the change log.

Look in this order:

1. Dashboards (RED metrics: rate, errors, duration; see the word list)
2. Recent deploys
3. Error tracking (new exception types)
4. Logs by request ID
5. Traces (where the time is spent in one request)
6. Dependency health (database, cache, queue, third parties)
7. Saturation (CPU, memory, connections, event-loop lag, disk)

5. Communicate

• Internal: send short, regular, factual updates, even when there is nothing new. For example: "Still investigating, rollback in progress, next update in 15 minutes."
• External: write a status page entry in plain language. Say what users see, what you are doing, and when the next update comes. Do not blame anyone, do not guess, and do not use internal jargon.
• Keep a single source of truth, so people do not interrupt the responders to ask for status.

6. Resolve and verify

Confirm the fix in metrics (error rate and latency back to normal). Do not stop at "it seems fine." Keep the extra monitoring on for a while. Then close the incident formally.

7. Blameless postmortem

A postmortem is a written review after the incident. Write it within a few days. A postmortem is a team learning document, not a report on who is guilty. Include:

• Impact (how long, how many users, how much money), the timeline, the root cause(s), and what went well, what went badly, and where you were lucky. Use "five whys" (ask "why?" again and again, about five times) to find causes. Prefer causes in the system over "human error".
• Action items with owners and dates. Put them in three groups:
  • prevent (stop this kind of bug)
  • detect (raise an alert earlier)
  • mitigate (roll back faster, or limit the damage)
• Blameless means you ask "how did our system allow this?" and not "who did this?" People who fear blame hide information. Then the next incident is worse.

Scenarios

1. "An endpoint's latency doubled overnight."

Symptom: p95 for GET /orders (95 out of 100 requests were faster than this time) went from 200 ms to 450 ms around 02:00. No alert fired, because the fixed limit was 1 s.

Diagnose:

1. What changed at 02:00? Check deploys, config, feature flags, scheduled jobs (backups, reports, cache warmers, VACUUM/ETL; see the word list), traffic and data growth.
2. Split the time with a trace: app handler, database, external calls, serialisation (turning data into JSON text) (Lesson 02).
3. If it is the database, look at the slow query logs and pg_stat_statements (a PostgreSQL extension that records how often and how long each query runs). Then run EXPLAIN (ANALYZE, BUFFERS) on the worst queries. Common nightly causes:
   • a batch job that competes for I/O (disk and network work) or locks (a database lock stops other queries from changing the same rows)
   • autovacuum (PostgreSQL's automatic cleanup of dead row versions)
   • a change in statistics that switches the query plan (the database picks a slower way to run the query)
   • an index that is no longer used
   • a table that grew past a size, so the cache no longer fits in memory
4. Compare slow and fast requests (large accounts and small ones, certain tenants).

Fix: stop or reschedule the competing job. Add or restore the index. Refresh the statistics (ANALYZE). Or add caching for the busy path.

Prevent: alert on p95 compared with the normal level, not only on a fixed limit. Mark deploys on dashboards. Run batch jobs on replicas or at quiet times, with resource limits. Add query-plan checks to code review.

2. "A bad deploy: error rate jumped from 0.1% to 12% five minutes after release."

Diagnose: the timing points at the release. Look in error tracking for the new exception. Compare versions in the logs (put the app version on every log line).

Fix: roll back at once (or switch the feature flag off). Confirm that the metrics recover. Then reproduce the bug in staging and fix it going forward.

Prevent: use a canary or progressive rollout (1%, then 10%, then 100%) with automatic rollback when the error rate goes over a limit. Run a smoke test (a quick check of the most basic functions) after each deploy. Use backward-compatible migrations, so rollback is safe. Put a feature flag around risky changes.

3. "Traffic suddenly increased 10x."

First, is it good traffic? It could be a marketing success or a viral post. It could be a bot or an attack. It could be a retry storm from your own clients (many clients retry at once, which makes things worse), or a loop in a mobile app release. Each case needs a different response.

Diagnose: look at who (IP ranges, user agents, API keys), what (which endpoints) and when it began. Check the autoscaling limits, database connections, cache hit ratio and queue depth.

Mitigate in this order:

1. Cache, and serve static or stale (slightly old) content (a CDN, which keeps copies of your files on servers near users, or stale-while-revalidate).
2. Scale out the stateless tiers. Check the total number of database connections as you scale. Use a pooler (see the word list) to keep that total below the database limit, so you do not just move the bottleneck to the database.
3. Rate-limit and shed load. Drop low-priority traffic, queue work that is not urgent, and return 429 with a Retry-After header. (Practical Design Problems, Lesson 02)
4. Degrade gracefully. Turn off recommendations, search suggestions and heavy reports.
5. Block abuse at the edge or with a WAF (web application firewall).

Prevent: run load tests at 5 to 10 times normal traffic. Plan autoscaling and spare capacity. Write a degradation plan (which features you can switch off, and in what order). Use CDN caching for public pages. Set capacity limits for each tenant.

4. "The database fails over and the application stays down."

Symptom: the primary database crashed and the replica became the new primary. But the app throws connection errors for 20 minutes.

Diagnose: the app may still use the old IP or cached DNS (DNS turns a name into an IP address, and the answer is cached) for the database. Connection pools may hold dead connections. The DNS TTL (how long the answer may be cached) may be high. Or the app does not retry. Check if the cluster endpoint changed.

Fix: restart or recycle the pools. Flush the DNS caches. Point the app to the new endpoint.

Prevent: use the provider's cluster endpoint (a stable name that always points to the current primary). Set pool settings (maxLifetime, connection checks, short idle timeouts). Retry temporary errors with backoff. Practise failovers on purpose (a "game day" is a planned drill where the team breaks something on purpose and practises the response).

5. "Everything slows down in a cascade; one slow dependency is taking the whole site down."

Diagnose: a downstream service became slow. Request threads and connections pile up while they wait for it. (A thread is one unit of work inside a process.) The pools run out. Then unrelated endpoints fail too. This is a cascading failure. Retries from clients make the load bigger. This is a retry storm.

Fix: open the circuit breaker (a switch that stops calls to a failing service), or turn off the feature that depends on it with a flag. Set short timeouts on the calls. Shed load. Scale the dependency if the problem is capacity.

Prevent: use timeouts everywhere, circuit breakers and bulkheads (a separate pool for each dependency, so one cannot use up everything). Use retry budgets (a cap on how many retries are allowed) with jitter (a random delay), fallbacks and load shedding. Keep a dependency map (a diagram of which service calls which). (Practical Design Problems, Lesson 07)

6. "A queue is backing up and consumers keep crashing."

Diagnose: look for these causes:

• A poison message (a message that always fails): it is retried forever and blocks the others.
• A consumer bug after a deploy.
• Slow processing downstream.
• A burst from a producer.

Check the age of the oldest message, the consumer error logs and the dead-letter queue. (A consumer is a program that reads and processes messages from the queue.)

Fix: move the poison message to the DLQ (dead-letter queue: a side queue for messages that keep failing). Roll back or fix the consumer. Scale the consumers if throughput is the problem. After the fix is deployed, replay the DLQ (send its messages through the queue again). Only do this when consumers are idempotent.

Prevent: set a maximum retry count and a DLQ with alerts. Make consumers idempotent (safe to run twice on the same message). Alert on queue age, not only on depth. Version the message schema (the agreed shape of the message).

7. "We discover a vulnerability in production (or a secret was leaked)."

Respond in this order:

1. Contain. Disable or limit the affected feature. Rotate the leaked secret or key at once. Revoke sessions or tokens if needed (force users to log in again). Block the attack path (a WAF rule).
2. Assess. What was exposed? Since when? Was it used by an attacker? Check audit logs (records of who did what), access logs and search for unusual activity. Keep the evidence.
3. Fix the root cause (a patch, validation, an authorization check) and deploy it.
4. Notify. Tell the security team and leadership first. If personal data was involved, the law may require you to tell the regulator and the users. The law sets deadlines. For example, GDPR (the EU data protection law) requires you to tell the regulator within 72 hours of becoming aware of the breach, and to tell the affected people without undue delay when the risk to them is high. Ask your legal team.
5. Postmortem and prevention.

Prevent: scan for secrets in CI (the automatic build and test system). Use short-lived credentials and least privilege (give each part only the access it needs). Scan dependencies. Run penetration tests (security experts try to break in, with your permission). Keep a written incident-response process and a security contact. Never "fix it quietly" without checking what was exposed.

8. "Customers report wrong or duplicated data."

Diagnose: a data-corruption incident is more serious than most outages. First stop more damage (pause the writer, turn off the feature). Find the blast radius (how much is affected) with queries: which rows, which time window, which code path. Then find the cause. It may be a race condition, a retry that is not idempotent, a bad migration (a change to the database structure) or a faulty backfill (a script that fills in old data).

Fix: correct the data with a reviewed script that can be undone. Run it on a copy first. Keep backups and point-in-time recovery (restoring the database to any chosen moment, not only to the last backup) ready. Check the result against a trusted source (the payments provider, the event log).

Prevent: use constraints (rules that the database enforces, such as "this value must be unique") and idempotency (Backend, Lesson 07). Rehearse migrations on production-sized copies. Run reconciliation jobs (they compare your records with another source, such as the payment provider, and report differences). Keep audit trails.

9. "A certificate or domain expired and half the clients cannot connect."

Fix first: renew the certificate and deploy it. Make sure every place that terminates TLS (servers, load balancers, CDN) loads the new one. If a domain expired, renew it at the registrar at once (most registrars keep a short grace period), then check auto-renew and the payment card. A certificate chain has a server certificate and one or more intermediate certificates. If an intermediate certificate is missing or expired, some clients (often older devices, which do not have it saved) fail while others work. That explains why only "half" fail.

Prevent: renew automatically (ACME is the standard protocol that lets software get and renew certificates by itself, for example with Let's Encrypt; or use managed certificates). Monitor expiry and alert at 30, 14 and 7 days. Keep a list of certificates, domains and API keys, with an owner for each.

10. "The third-party payment provider is down."

Do this: switch checkout to a fallback (a second provider, or "pay later", or queue the order). Show an honest message. Queue the orders and retry with idempotency keys, so no one is charged twice. Pause marketing spend. Watch the provider's status page. Reconcile when it comes back.

Prevent: put an abstraction layer over the providers. Add circuit breakers and graceful degradation (the product keeps its core function and switches off the extras). Make payment flows idempotent. Keep a runbook (a step-by-step guide for this failure).

11. "You are on call at 3 a.m. and the page says 'high error rate'. What do you do in the first 10 minutes?"

1. Acknowledge the page (the alert that woke you) and check the impact. Is it real? Look at the dashboards or a synthetic check (a fake user test that runs on a schedule).
2. What changed? Look at the last deploy and the flags. Roll back if they match the problem.
3. Find the scope. All users, or only some region, tenant or endpoint? Which dependency is red?
4. Mitigate. Roll back, restart a stuck instance, scale, fail over, or switch a flag off.
5. Escalate early if the problem is bigger than you or outside your area. Waking a colleague is not a failure.
6. Communicate in the channel and on the status page.
7. Write down what you did, for the postmortem.

How to answer incident questions in an interview

Structure your answer as Detect, Mitigate, Diagnose, Communicate, Resolve, Learn. Then mention these senior signals:

• You roll back first.
• You look at what changed.
• You find the scope of the impact before you dive deep.
• You communicate.
• You run a blameless postmortem with prevent, detect and mitigate action items.

If they ask "tell me about a real incident", use the STAR format (Situation, Task, Action, Result) and give details. Say the symptom, your role, the decision you made and why, the outcome, and what changed afterwards.

Common mistakes

1. Debugging before mitigating, so users suffer while you investigate.
2. No clear incident commander, which causes repeated or conflicting actions.
3. Changing several things at once, so you cannot tell what worked.
4. Staying silent towards stakeholders and customers.
5. Blaming people instead of fixing systems.
6. Declaring victory without checking metrics.
7. No follow-through on postmortem action items.

Interview questions

Key takeaways

• Mitigate first (roll back, flag off, scale, shed, fail over). Investigate second.
• Always ask "what changed?" and find the scope of the impact before you dive deep.
• Use roles, a timeline and regular communication (inside and outside the team).
• Check the recovery in metrics. Then run a blameless postmortem with prevent, detect and mitigate actions.
• Prepare for incidents in advance: rollbacks, flags, canaries, timeouts, circuit breakers, DLQs, runbooks, and alerts that compare against the normal level.

What's next

Lesson 11: Backend and API Scenario Bank. A wider set of "what would you do?" questions for the server side. Each one has diagnose, fix and prevent parts.