Reliability Patterns: Timeouts, Retries, Circuit Breakers and Graceful Degradation

Last lesson ended with a fictional post-mortem. A payments service had a two-minute slowdown, and a bad retry setting turned it into a 37-minute incident. Everyone nodded at the action items: "add backoff", "add a circuit breaker", "shed load". This lesson makes those words into running code, so you can see them work.

We will do it the honest way. For each pattern I first show the problem with real output, and then the fix with real output. We inject faults on purpose: a server that hangs, a server that fails for one second, a dependency that goes down for two seconds. This is called fault injection, and it is the only way to know that a safety feature really works. A circuit breaker you have never seen open is a guess, not a feature.

This is lesson 7 of a nine-lesson series. The incident that motivates it is in Incidents, On-Call and Blameless Post-Mortems. The numbers we use to judge success (error ratio, latency percentiles) come from SLIs, SLOs and Error Budgets, and the way to see the problems live is in Metrics and Prometheus. The next and last lesson, Observability and SRE Interview Questions With Model Answers, collects the questions you may be asked about all of this.

Words used in this lesson

• Dependency is something your code calls and needs: another service, a database, a cache, a payment provider.
• Failure mode is a way a thing can go wrong. A dependency can be down, slow, hung, wrong, or overloaded. Each needs a different answer.
• Timeout is a time limit. If the answer does not arrive in time, you stop waiting and treat it as a failure.
• Deadline is a timeout written as a clock time ("must finish by 14:00:01.200"), so it can be shared across many calls.
• Retry means trying the same call again after a failure.
• Backoff means waiting between retries, usually longer each time (1, 2, 4, 8 steps). Exponential backoff doubles the wait each time.
• Jitter is a bit of randomness added to the wait, so many clients do not all retry at the same moment.
• Thundering herd is a crowd of clients that all act at the same moment, for example all retrying at 100 ms after a failure.
• Retry storm (or retry amplification) happens when retries multiply the load on a struggling service so it cannot recover.
• Retry budget is a cap on how much extra load retries may add, for example "retries may be at most 10% of normal traffic".
• Idempotent means doing it twice has the same effect as doing it once. Reading a page is idempotent. Charging a card is not, unless you add a key.
• Idempotency key is a unique id sent with a request, so the server can notice a repeat and not do the work twice.
• Circuit breaker is a switch around a dependency. When too many calls fail, it opens and fails new calls instantly, for a while, instead of waiting on a broken thing.
• Half-open is the careful state after the wait: the breaker lets one trial call through to see if the dependency is back.
• Bulkhead is a limit on how many resources one dependency may use, so a failure in one cannot use up everything. The word comes from the walls inside a ship that stop one leak from sinking it.
• Load shedding means refusing some work on purpose when you are overloaded, quickly, so the work you accept still goes well.
• Graceful degradation means a smaller but useful answer when something fails (an old cached price, an empty recommendations list), instead of an error page.
• Fallback is the "plan B" answer used when the main call fails.
• Cascading failure is when one failure causes the next, and the next, until much of the system is down.
• Fault injection means causing failures on purpose, in a test, to check that your safety code works.

The shape of the problem: slow is worse than down

Beginners fear a dependency that is down. A down dependency is easy: the connection is refused in a millisecond, your code sees an error, and it can react. The dangerous one is the dependency that is slow or hung. It accepts your request and then says nothing.

Why is that worse? Because your code is now waiting, and waiting costs something. A waiting request holds a socket, some memory, a slot in a worker pool, and a user's patience. If a hundred requests per second arrive and each waits 30 seconds, you now hold 3,000 waiting requests. Your own service runs out of room, becomes slow, and now your callers are waiting on you. This is how a failure spreads upwards, step by step. That spread is a cascading failure.

So the first job is not clever. It is to put a limit on waiting.

Pattern 1: Timeouts

A timeout says "I will wait this long, no more." Let us see what happens without one. In a-timeout.mjs I start a toy server that accepts requests and never answers (an injected "hang" fault). Then I call it twice: once with no timeout, once with a 300 ms timeout.

const server = http.createServer(() => { /* never respond */ });// ...// 1) No timeout. I stop watching after 2 s only so the demo can end.const watcher = AbortSignal.timeout(2000);try { await fetch(url, { signal: watcher }); } catch (e) { /* ... */ }// 2) With a timeout of 300 ms per attempt.try { await fetch(url, { signal: AbortSignal.timeout(300) }); } catch (e) { /* ... */ }

Real output:

no timeout set: still waiting after 2004 ms (I stopped it myself: TimeoutError)with timeout 300: failed after 301 ms, error name = TimeoutError

Without the timeout, the call would wait until the server or the network gave up, which can be minutes. With AbortSignal.timeout(300), the call fails after 301 ms. AbortSignal.timeout() is built into Node 22 and into modern browsers, and it works with fetch directly. For older libraries that do not take a signal, a pair new AbortController() and setTimeout(() => controller.abort(), ms) does the same thing.

How long should a timeout be?

Do not guess. Look at your data from the metrics lesson. A good starting rule: set the timeout a little above the p99 latency of a healthy call. If the healthy p99 is 150 ms, a timeout of 300 to 500 ms is reasonable. Shorter than p99 and you turn healthy slow calls into failures. Much longer than p99 and the timeout does not protect you.

Also use two kinds of timeout where your tool allows it:

• a connect timeout (how long to wait for a connection to be made; short, like 100 to 500 ms), and
• a total timeout for the whole request including the reply.

Deadlines: one budget for the whole request

Here is a trap. A request enters your system with a promise to the user: "you will get an answer within 1 second". It passes through three services. If each has its own 1-second timeout, the total can be 3 seconds, and the user left long ago. Everyone does useless work.

The fix is a deadline: the first service decides "this request must be done by time T", and passes T down. Every service uses time left = T - now as its limit and stops early if the time is gone. (gRPC works like this: the README for its retry design says that call deadlines apply to the whole chain of retries and hedged calls, and once the deadline has passed the call fails whatever else is happening.) A tiny version, h-deadline.mjs: three attempts at most, each attempt limited to 400 ms, but never past the total of 1,000 ms.

async function callWithDeadline(totalMs, perAttemptMs, doCall) {  const deadline = Date.now() + totalMs; let n = 0;  while (true) {    const left = deadline - Date.now();    if (left <= 0) return `gave up after ${n} attempts`;    n++;    const ac = new AbortController();    const timer = setTimeout(() => ac.abort(), Math.min(perAttemptMs, left));    try { return await doCall(ac.signal) + ` on attempt ${n}`; }    catch { await sleep(Math.min(50 * 2 ** (n - 1) * Math.random(), Math.max(0, deadline - Date.now()))); }    finally { clearTimeout(timer); }  }}

Output (a dependency that always hangs, then one that works from the third try):

gave up after 3 attempts, 1000 ms used (wall clock 1001 ms)ok on attempt 3

The call never takes more than the total budget, no matter how many attempts are allowed. That is the point of a deadline.

Pattern 2: Retries (and why they are dangerous)

Many failures are brief: a network blip, a server restarting, one overloaded replica. A retry turns such a failure into a success for the user. So retries are good. But a retry is also more load on a service that is already having trouble. A retry is a bet that things will be better in a moment. Used badly, it is a weapon pointed at your own dependency.

What is safe to retry?

Before any code, ask two questions.

1. Is the failure the kind that may pass? A timeout, a connection reset, an HTTP 503 or 429 may pass. A 400 (your request is bad) or 404 will fail the same way every time. Do not retry those. gRPC's design lets you list exactly which status codes are retryable, which is a good habit.
2. Is the operation idempotent? Retrying "get user 5" is harmless. Retrying "charge $50" may charge twice, because a timeout does not mean the work was not done. Often the server did the work, and only the reply was lost or slow.

Let us prove the second point, because it is the most expensive mistake in this lesson. In f-idem.mjs the server takes 150 ms to charge, but the client gives up after 100 ms and retries up to three times. The first call is still running on the server when the retry arrives.

key: false stored before work: false -> customer charged 3 time(s): [50,50,50]key: true  stored before work: false -> customer charged 2 time(s): [50,50]key: true  stored before work: true  -> customer charged 1 time(s): [50]

Read the three lines:

• No key: the customer is charged three times.
• Key, but the server saves the key only after finishing the work: still charged twice, because the retry arrives while the first call is still running, and nothing has been saved yet. This is a real bug that people write.
• Key, saved before the work starts (we store the in-progress promise under the key, so the repeat waits for the same result): charged once.

An idempotency key is a unique id (for example order-1001) that the client creates once per user action and sends on every attempt. The server must record "I have seen this key" at the start, and return the same answer to repeats. Payment providers use this idea in their public APIs for exactly this reason. For the HTTP side of this idea, see the REST lesson in the backend series: REST API Design Best Practices.

Retry with nothing else: the storm

Now the main demo. This reproduces the shape of last lesson's post-mortem, at a small scale.

The setup (server.mjs and b-storm.mjs):

• A toy "payments" server in its own process. Every request costs it 2.5 ms of CPU (parsing, logging, auth: I used a busy loop to model that), even if it ends up refusing the request. It can do at most 12 requests at once and answers "503 busy" above that.
• Fault injection: between 1.5 s and 2.5 s after start, the server answers 503 to everything. Think "a database replica is being promoted".
• Clients: about 300 users per second for 8 seconds. Each attempt has a 250 ms timeout.
• Three client policies: none (no retries), immediate (up to 5 attempts, no waiting) and backoff (up to 3 attempts, exponential backoff with full jitter, plus a retry budget).

The immediate policy is the bug from the post-mortem:

let ok = await attempt();if (!ok && POLICY === 'immediate')  for (let k = 0; k < 4 && !ok; k++) ok = await attempt();   // 5 attempts, no wait

First, no retries. The server fails for one second, and users feel exactly that second:

policy: none   (server blip: 1.5s to 2.5s, 503 on everything)second | users | attempts | attempts/user | users served     0 |   291 |      291 |          1.00 |       100%     1 |   294 |      294 |          1.00 |        49%     2 |   297 |      297 |          1.00 |        51%     3 |   279 |      279 |          1.00 |       100%     4 |   291 |      291 |          1.00 |       100%     5 |   294 |      294 |          1.00 |       100%     6 |   294 |      294 |          1.00 |       100%     7 |   297 |      297 |          1.00 |       100%server saw: {"received":2337,"ok":2042,"shed":1,"injected":294}

The blip is one second long (it falls half in second 1 and half in second 2, so about half of the users in each of those seconds failed). By second 3 everything is 100% again. That is a healthy failure shape: the pain lasts as long as the fault.

Now immediate retries, same fault, same traffic:

policy: immediate   (server blip: 1.5s to 2.5s, 503 on everything)second | users | attempts | attempts/user | users served     0 |   282 |      282 |          1.00 |       100%     1 |   294 |      487 |          1.66 |        56%     2 |   288 |     1124 |          3.90 |        56%     3 |   291 |     1050 |          3.61 |        72%     4 |   282 |      994 |          3.52 |        72%     5 |   294 |      919 |          3.13 |        79%     6 |   273 |      852 |          3.12 |        74%     7 |   294 |      832 |          2.83 |        96%server saw: {"received":3463,"ok":2366,"shed":725,"injected":365}

Look at the column "attempts/user". It rose to about 3.9. Our traffic to the server went from 300 requests per second to more than 1,100. The server is only designed for about 400 requests per second at 2.5 ms each, so it spends its time rejecting and timing out. The fault ended at second 2.5, but the pain went on for about five more seconds: second 3 to second 6 still have 72% to 79% of users served, and "shed" is 725, meaning the server refused 725 requests after the real fault was gone. The retries themselves kept the server overloaded. That is a retry storm. (It slowly dies in my demo because the users stop coming at second 8 and the backlog drains. In a real system with steady traffic, it can stay stuck for a long time; people sometimes call this a metastable failure: the trigger is gone, but the system stays broken because of its own reaction.)

Finally the backoff policy:

// 3 attempts, backoff + full jitter + budgetfor (let k = 0; k < 2 && !ok; k++) {  if (tokens < 1) break; tokens--;                 // retry budget  await sleep(Math.random() * Math.min(2000, 100 * 2 ** (k + 1)));  ok = await attempt();}

policy: backoff   (server blip: 1.5s to 2.5s, 503 on everything)second | users | attempts | attempts/user | users served     0 |   294 |      294 |          1.00 |       100%     1 |   294 |      314 |          1.07 |        49%     2 |   294 |      294 |          1.00 |        51%     3 |   297 |      297 |          1.00 |       100%     4 |   288 |      288 |          1.00 |       100%     5 |   285 |      285 |          1.00 |       100%     6 |   270 |      270 |          1.00 |       100%     7 |   294 |      294 |          1.00 |       100%server saw: {"received":2336,"ok":2021,"shed":0,"injected":315}

The extra load is tiny (1.07 attempts per user at worst), the server is never overloaded (shed is 0), and recovery is instant after the fault. Notice an honest detail: during the blip itself, the backoff policy did not save the users either (49% and 51%, the same as none). Retries cannot fix a fault longer than the retry window. Their job is to fix short faults and not to make long faults worse.

Why was the budget needed here? Because the fault lasted a whole second, and every user would have made their two extra attempts. The budget (about one retry token per 10 successes, up to 20 saved) cut that down to a trickle. Retries are for rare blips, so a retry budget enforces the idea "retries are rare" in code.

Backoff and jitter

Backoff means waiting longer between tries: 100 ms, 200 ms, 400 ms and so on, up to a cap. It gives the dependency time to recover. But backoff alone has a hidden problem. Suppose 200 clients all failed at the same instant (a server restarted). With plain exponential backoff, all 200 wait exactly 100 ms and retry at the same moment, then all wait 200 ms and retry together. The herd stays a herd. c-herd.mjs counts the retries landing in each 10 ms slice:

fixed 100ms, no jitter       retries=1000  busiest 10ms bucket=200  buckets used=5  span=510 msexponential, no jitter       retries=1000  busiest 10ms bucket=200  buckets used=5  span=3110 msexponential, full jitter     retries=1000  busiest 10ms bucket=35  buckets used=211  span=2830 ms

Without jitter, all 200 retries land in one 10 ms slice on every round, even with exponential backoff. With full jitter (wait a random time between 0 and the current backoff limit), the busiest slice has 35 retries and the load is spread over 211 slices. The total number of retries is the same (1,000). Only the shape changed, from five spikes to a smooth spread.

The formula I used: delay = random() * min(cap, base * 2^attempt). The gRPC retry design uses a related rule. Its delay is min(initialBackoff * multiplier^(n-1), maxBackoff) and then it applies a random jitter of plus or minus 20% to avoid many clients hitting the server at the same time. Both are fine. The key is: some randomness.

Retries multiply across layers

One more piece of arithmetic, and it is scary. Suppose a request passes through service A, which calls B, which calls C, which calls D. Each layer retries up to 3 times. How many calls reach D when D is down?

1 layers x 3 attempts = 3 calls to the bottom service2 layers x 3 attempts = 9 calls to the bottom service3 layers x 3 attempts = 27 calls to the bottom service4 layers x 3 attempts = 81 calls to the bottom service

One user click became 81 calls on the bottom service. That is why many teams retry in one layer only (usually the one closest to the user, or the one that knows if the operation is safe to repeat), and pass a "do not retry" hint down. The lesson of the retry budget is also in the gRPC design: gRPC clients keep a token count per server. Every failed call takes 1 token, every successful call gives back a small fraction (for example 0.1), and when the tokens fall to half of the maximum or below, retries are simply not made. That is the same idea as my tokens variable above, built into a library.

A retry checklist

• Retry only failures that may pass, and only idempotent calls (or calls with an idempotency key).
• Few attempts (2 or 3 in total is common), never "forever".
• Exponential backoff with jitter and a cap.
• A total deadline that includes all the attempts.
• A retry budget or throttle, so retries stay a small share of traffic.
• Retry in one layer, not every layer.
• Measure it: export "attempts per request" and "retries per second" as metrics. In the post-mortem, the missing panel was exactly "outgoing calls per incoming request".

Pattern 3: Circuit breakers

Timeouts and retries are about one call. A circuit breaker is about the dependency as a whole. The idea comes from the electrical breaker in your house: when too much current flows, the breaker trips so the wires do not burn. In software, when too many calls fail, the breaker opens and new calls fail at once without touching the broken thing. This protects two parties. Your callers get an instant answer instead of a slow one, and the dependency gets a rest, so it can recover.

A breaker has three states:

            too many failures                  wait time is over CLOSED  ------------------------>  OPEN  ------------------------>  HALF-OPEN (calls go through)                (calls fail fast)             (let one test call through)    ^                                 ^                                   |    |            test call fails      |                                   |    |         <-----------------------+-----------------------------------+    +------------------------- test call succeeds -----------------------+

The README of opossum describes the same life cycle in its words: once the circuit has opened, a timer based on resetTimeout starts. When it ends, the breaker goes to halfOpen, and the next call is allowed to run. If it succeeds the circuit closes. If it fails or times out, the circuit goes straight back to open.

A breaker in 35 lines

Here is breaker.mjs, my own small version. It counts consecutive failures, which is the simplest rule. Real libraries use a failure percentage over a rolling window, which is better, as we see at the end.

export class Breaker {  constructor({ failures = 5, resetMs = 1000, onState = () => {} } = {}) {    Object.assign(this, { failures, resetMs, onState, state: 'closed', count: 0, openedAt: 0, probing: false });  }  #set(s) { if (s !== this.state) { this.state = s; this.onState(s); } }  async call(fn) {    if (this.state === 'open') {      if (Date.now() - this.openedAt < this.resetMs) throw new Error('circuit open');      this.#set('half-open');    }    if (this.state === 'half-open') {      if (this.probing) throw new Error('circuit open');   // only one probe at a time      this.probing = true;    }    try {      const v = await fn();      this.count = 0; this.probing = false; this.#set('closed');      return v;    } catch (e) {      this.probing = false;      this.count++;      if (this.state === 'half-open' || this.count >= this.failures) { this.openedAt = Date.now(); this.#set('open'); }      throw e;    }  }}

Test it with fault injection

d-breaker.mjs calls a "price service" every 50 ms for about 5 seconds. The injected fault: between 1,000 ms and 3,000 ms the price service hangs. Each call has a 100 ms timeout. We run the loop twice. Both runs use a fallback: if the live call fails, return the last good price (marked stale). This is graceful degradation, which we cover below. The only difference between the runs is the breaker.

WITHOUT breaker (timeout 100 ms + stale fallback only):  dependency calls: 100   live: 80  stale-fallback: 20  failed: 0  mean latency during outage: 60 ms   worst: 101 msWITH breaker (5 failures opens it, 700 ms before a probe):  [1522 ms] breaker -> open  [2228 ms] breaker -> half-open  [2328 ms] breaker -> open  [3045 ms] breaker -> half-open  [3050 ms] breaker -> closed  dependency calls: 72   live: 66  stale-fallback: 34  failed: 0  mean latency during outage: 17 ms   worst: 101 ms

Read it like a story:

• At 1,522 ms the breaker opened, after five timeouts in a row. From then on calls fail in about a millisecond instead of 100 ms.
• At 2,228 ms it became half-open, ready for one test call. That test hit the dependency, which was still hung, so it timed out at 2,328 ms and the breaker went straight back to open. That is the half-open rule working: one failed test, back to waiting.
• At 3,045 ms the next test happened after the fault had ended at 3,000 ms. It worked, and the breaker closed at 3,050 ms. Normal service resumed without anyone touching anything.
• Dependency calls dropped from 100 to 72. The breaker kept 28 calls away from a broken service.
• Mean latency during the outage fell from 60 ms to 17 ms. The rest of the delay in the "with" run is the five calls that had to time out before the breaker opened, and the test call.

Two honest limits. First, the dependency here is a function in the same process, and the run lasts 5 seconds, so the counts are small. The gain looks modest only because the demo is small: in production the saved calls would be thousands per second. Second, because I used Promise.race for the timeout, the abandoned "hung" calls keep sleeping in the background. A real HTTP client with AbortSignal.timeout would close those connections.

The same thing with a real library

You should not usually write your own breaker. Here is the same idea with opossum 10.0.0 from npm (script op/op.mjs): a dependency that hangs for calls 10 to 24 of 40, a 100 ms timeout, a failure threshold of 50%, a reset time of 400 ms, and a fallback that returns 'stale'.

const b = new CircuitBreaker(dep, { timeout: 100, errorThresholdPercentage: 50, resetTimeout: 400, volumeThreshold: 5 });b.fallback(() => 'stale');

event: openevent: halfOpenevent: close{ live: 22, stale: 18 } real dependency calls: 33 stats.fires 40 rejects 7 timeouts 11

All 40 user calls got an answer (22 live, 18 stale). Only 33 of the 40 calls reached the dependency; 7 were rejected by the open breaker; 11 timed out before it opened. The events open, halfOpen and close match the state picture above. The README explains that errorThresholdPercentage is compared with the error rate computed over the calls in the rolling window, and that nothing opens until at least volumeThreshold calls have been made. That rule avoids opening the circuit because of "1 failure out of 1 call".

Choosing breaker settings

• Failure threshold: a percentage over a window (such as 50% over the last 10 seconds) is better than "N in a row", because it works for both low and high traffic. Add a minimum number of calls, so tiny samples do not trip it.
• What counts as a failure: timeouts and 5xx errors, yes. A 404 or a validation error (4xx) means "the dependency answered correctly", so do not count them.
• Reset time: long enough for a real recovery (seconds to tens of seconds), not so long that you stay open after it is back. Some libraries grow the wait for repeated failed tests.
• One breaker per dependency (and sometimes per endpoint), not one for the whole app. A breaker around "all HTTP calls" would open for everyone when one service is down.
• Make it visible. Export the state as a gauge (0 closed, 1 half-open, 2 open) and count rejects. A breaker that opens silently becomes a mystery during an incident. An alert for "breaker open for 5 minutes" is useful.

A history note: Hystrix, the Netflix library that made this pattern famous in Java, says in its own README that it is in maintenance mode and no longer actively developed. It points to more adaptive approaches (limits that react to measured performance) and to resilience4j for new work. The pattern did not die; the library did. In Node, opossum is a common choice.

Pattern 4: Graceful degradation and fallbacks

When a dependency fails, the options are:

1. Show an error. Honest, but the worst for the user.
2. Show something smaller or older. This is graceful degradation.
3. Hide the part. If recommendations fail, show the page without them.

Examples of fallbacks, from safest to riskiest:

The skill is deciding, for each dependency, what is the right smaller answer, and writing it down before the incident. That is a product decision as much as an engineering one: "if prices are unavailable, may we show yesterday's?" Ask the product owner in daylight.

In d-breaker.mjs, the fallback is two lines: remember the last good value and return it on failure. The result in both runs: failed: 0. Users always got a price. 34 of them got an older one during the outage.

Pattern 5: Bulkheads

On a ship, walls inside the hull split it into rooms. A leak floods one room, not the whole ship. In software, a bulkhead is a separate limit for each dependency, so one slow dependency can use up only its own share of workers, sockets or memory.

The failure it prevents: your service has one shared pool of 10 slots for outgoing calls. The recommendations service becomes very slow (2 seconds per call). Soon all 10 slots hold recommendations calls, and your healthy payments calls have nowhere to run. A non-critical feature has taken down the critical one.

e-bulkhead.mjs reproduces this. For 3 seconds it sends 20 recommendations calls and 20 payments calls per second. Recommendations take 2 seconds (fault injected). Payments take 20 ms (healthy). Callers wait at most 500 ms for a free slot before being rejected.

SHARED pool of 10 for both dependencies:  payments: ok 9/60, failed 51, p50 20 ms, p95 21 ms  recommendations: rejected 40, accepted (slow, 2 s each) 20BULKHEADS (payments pool 5, recommendations pool 5, 0 ms queue):  payments: ok 60/60, failed 0, p50 20 ms, p95 21 ms  recommendations: rejected 50, accepted (slow, 2 s each) 10

With a shared pool, only 9 of 60 payments calls worked, even though the payments service was perfectly healthy. The slow recommendations calls held the slots. With separate pools, all 60 payments calls worked, and recommendations were rejected quickly when their own 5 slots were full. The latency of the successful payments calls is the same in both runs (20 ms), which shows the trap for monitoring: the payments latency looked fine in the shared case; the damage is in the failed count. Always look at both errors and latency.

The resilience4j README names this pattern too: it separates a semaphore-based bulkhead (limit the number of concurrent calls) and a thread-pool bulkhead (separate threads), and it warns to use a separate bulkhead for each downstream service and not one shared object. In Node there are no threads to count, so you limit concurrency with a counter, a semaphore or a small queue library, as I did with the Pool class.

Bulkheads exist at larger scales too: separate connection pools per dependency, separate worker processes for risky jobs, separate Kubernetes deployments for background work and user traffic, and "cells" that serve only a slice of customers.

Pattern 6: Load shedding and backpressure

A service has a maximum speed. What should it do when asked for more? There are only two options: queue the extra work, or refuse it. Queueing sounds polite. Let us see what it does. g-shed.mjs models a service that can handle 4 jobs at once, 20 ms each, so 200 jobs per second at most. We send 400 jobs per second for 2 seconds (800 jobs). Version one has an unlimited queue. Version two refuses new work when 8 are already waiting (it "sheds" them with an instant 503).

no limit (queue forever)   served 800  shed   0  p50 1188 ms  p99 2314 msshed above 8 waiting       served 360  shed 440  p50   58 ms  p99   62 ms

With the unlimited queue, everybody is served, and everybody waits a long time: the median job waits 1.2 seconds, and the slowest wait 2.3 seconds. In real life, clients would time out at that point and retry, adding even more work. Many of those answers would come too late for anyone to use, which is pure wasted effort.

With shedding, 440 jobs get an instant "no", and the 360 accepted jobs finish in about 60 ms. The service delivered about the same real throughput (360 vs the 400 it can do in 2 seconds), and the accepted users had a good experience. The refused clients know at once, so they can use a fallback or back off.

A shedding response should be well-behaved:

• Return 503 Service Unavailable (or 429 Too Many Requests if you are limiting a particular client) as fast as possible.
• Add a Retry-After header when you can tell clients when to try again. My toy server in the storm demo sends retry-after: 1.
• Prefer to shed less important work first: background jobs before user clicks, free-tier before paying customers, reads of rarely-used data before checkout.
• Shed early, before spending CPU. In my storm demo, the CPU cost of rejecting was exactly why the server stayed overloaded. Rejecting at the front door (a load balancer, an API gateway, the first line of the handler) is far cheaper than at the bottom.

This is also what the post-mortem's action item 4 meant: "payments-svc sheds load above a concurrency limit and returns 503 with Retry-After".

Backpressure is the sibling idea: a slow consumer tells the producer to slow down, instead of letting data pile up in memory. In Node, streams do this for you if you respect write() returning false. In HTTP services, a 429 or 503 with Retry-After is how a server pushes back on clients.

Putting the patterns together

Each pattern covers a different hole. A short map:

In code, the order of layers around one call is usually: bulkhead (do I have a slot?), then circuit breaker (is the dependency considered healthy?), then retry with backoff (inside the total deadline), then timeout for each attempt. And around all of it, a fallback for the final failure. Libraries such as resilience4j and opossum let you compose these.

// Sketch only. A real library handles the details.async function getPrice(sku) {  return withFallback(                           // 5. plan B: last known price    () => bulkhead.run(                          // 1. limit concurrent calls      () => breaker.call(                        // 2. fail fast when broken        () => retryWithBackoff(                  // 3. few retries, jittered, in the deadline          (signal) => fetch(url, { signal })     // 4. per-attempt timeout via the signal        ))),    () => cache.get(sku));}

Test it: fault injection and game days

All of the above is just opinion until you break things on purpose. Fault injection means making a failure happen in a controlled way. At small scale, you did it already in this lesson: a hung server, a one-second 503, a slow dependency. Ways to do the same for your own service:

• In tests: a fake dependency with a switch ("fail", "hang", "slow") like my server.mjs. Write tests such as "when payments hangs for 2 seconds, checkout returns within 500 ms with a clear error".
• With a proxy: tools such as Toxiproxy sit between your service and a dependency and add latency, resets or timeouts. I did not run it for this lesson, so I only name it as an example of the kind of tool to look for.
• In staging or production, carefully: game days, where the team plans a failure (kill a replica, block a dependency), watches the dashboards, and checks that the breaker opens, the alert fires and the fallback works. Start small, with a clear way to stop, and tell people.

Three things to check in each experiment: (1) did the system protect itself (breaker opened, load stayed bounded)? (2) did we notice (alert, dashboard panel)? (3) did it recover by itself after the fault ended? Write the result down, as you would for a real incident.

I should say that first result openly: with a free-to-reject server, even 5 immediate retries per user did no lasting harm. The reason retries are dangerous is not that they are always bad. It is that they are bad exactly when the system is already close to its limit, which is also when you most need it to survive.

Common mistakes

• No timeout at all. The default of many HTTP clients is "wait forever" or "minutes". Always set one.
• Timeout longer than the caller's timeout. If you wait 10 s and your caller gave up at 2 s, you do work nobody will read.
• Retrying everything. Retrying a 400, or a non-idempotent POST without a key.
• Retrying without backoff or jitter. The exact bug in last lesson's post-mortem.
• Retries at every layer. 3 x 3 x 3 = 27.
• One breaker for everything, or one that never gets tested.
• Fallbacks that hide problems. Always count and alert on fallback use.
• Shared pools. A non-critical dependency starving a critical one.
• Queues with no limit. An unlimited queue is a delay with no upper bound, and often a memory leak too.
• Only testing success. If you have not injected the failure, you do not know what your code does.
• Treating resilience code as finished. Settings that were right at 100 requests per second may be wrong at 10,000. Review them with your latency data now and then.

Try it

1. In a-timeout.mjs style, write a small server that responds after 1 second. Call it with AbortSignal.timeout(500) and with AbortSignal.timeout(1500). What do you see in each case? Which one would you choose in production if the healthy p99 is 150 ms?
2. A client retries up to 4 times with a delay of 100 * 2^attempt ms and no jitter. 1,000 clients fail at time 0. At which times do the retries land, and how many land at each time? What changes if you use full jitter?
3. Three services are chained: A calls B, B calls C, C calls D. Each retries up to 3 times (so 3 attempts in total). D goes down. How many calls does D get for one user click? If only A retries, how many?
4. In f-idem.mjs, move the seen.set to run after the work and explain why two charges still happen. Then fix it.
5. Your recommendation service has a 2-second latency. Your page calls it, and also a payments service, from one shared pool of 20 connections. Design the bulkheads: how many slots for each, and what do you do when the recommendation pool is full?

Answers

1. With 500 ms against a 1-second server you get a TimeoutError after about 500 ms. With 1,500 ms you get an answer after about 1 second. In production with a healthy p99 of 150 ms, something like 300 to 500 ms is a sensible per-attempt timeout: above the healthy tail, and far below "forever".
2. Delays are 200, 400, 800 and 1,600 ms (100 * 2^1 to 100 * 2^4). Without jitter, every client retries at 200, then 600, then 1,400, then 3,000 ms, since the waits add up, so you get four spikes of up to 1,000 retries each. With full jitter, the retries spread over the whole window, so the busiest moment is a small fraction of 1,000 (as in the herd demo, 35 vs 200 per 10 ms for 200 clients).
3. With retries in all three layers: 3 x 3 x 3 = 27 calls reach D. If only A retries: 3 calls reach D (A's retries pass through B and C once each, assuming B and C do not retry).
4. If the key is saved only after the work finishes, a repeat that arrives while the first call is still running finds no key and starts a second charge. Fix: save the key (or the in-progress promise) at the start, as in the third line of the output: charged once.
5. One reasonable design: 12 slots for payments (critical, healthy) and 8 for recommendations (optional), with a very short queue wait on recommendations (or none). When the recommendations pool is full, do not wait: return an empty list or a fixed "popular items" list, count it as a fallback, and show the page. The exact numbers should come from measured traffic (how many concurrent payments calls you need at peak, plus some margin), not from my guess.

Interview questions

Key takeaways

• A slow dependency is more dangerous than a dead one. Put a timeout on every call, and share a deadline across the whole request.
• Retry only failures that may pass, only idempotent operations (or with an idempotency key), a few times, with exponential backoff and jitter, inside a deadline, with a retry budget.
• Retries multiply across layers (3 x 3 x 3 = 27). Retry in one layer and measure attempts per request.
• A circuit breaker fails fast when a dependency is broken, lets a trial call through after a wait, and closes by itself. One breaker per dependency, with a minimum volume, and make its state visible.
• A bulkhead gives each dependency its own limit so a slow optional feature cannot starve a critical one.
• Load shedding says no early, so accepted work stays fast. An unlimited queue only turns overload into delay.
• Graceful degradation needs plan B decided in advance, marked as degraded, and counted so it cannot hide an outage.
• Test with fault injection: hangs, slowness, flapping and partial failure. A kind test gives false comfort.

What's next

Next: Observability and SRE Interview Questions With Model Answers. This is the last lesson of the series. It collects about seventeen questions on monitoring, the three pillars, SLOs and error budgets, alerting, incidents and the reliability patterns from this lesson, each with a model answer, so you can practise saying it out loud. Practise now: pick one pattern from this lesson, run its script, change one number (the timeout, the breaker threshold, the pool size), and predict the output before you run it again. For where these ideas fit with deployment and operations, see the DevOps pages.