Timeouts, Retries & Backoff with Jitter

Part 1: Timeouts

Every network call needs a timeout. Many libraries and HTTP clients have no timeout, or a very long one, by default. Always set timeouts yourself.

Types of timeouts:

• Connect timeout: how long to wait to open the connection. It is usually short (like 100 ms–1 s), because a healthy server accepts quickly.
• Read (response) timeout: how long to wait for data after the connection is open. In some tools, such as Python requests, this is the wait between pieces of data, not the whole download.
• Total (request) timeout: the overall limit for the whole call, including retries if you count them together.
• Idle timeouts close keep-alive connections (connections kept open for reuse) that nobody uses. Pool acquire timeouts limit how long to wait for a free connection from a pool (a ready-made set of connections that many requests share).

// Node.js fetch with a timeout using AbortControllerconst res = await fetch(url, { signal: AbortSignal.timeout(300) });  // 300 ms total

# Python requests: (connect timeout, read timeout)requests.get(url, timeout=(0.5, 2.0))

Choosing timeout values

Use measured latency, not guesses:

• Look at the dependency's p99 or p99.9 latency (post 8). Latency is the time a call takes. p99 is the time that 99 out of 100 calls beat. If p99 is 120 ms, a timeout of about 300–500 ms leaves room for normal slow calls and still cuts off calls that are really stuck.
• Too short: you time out on requests that are slow but normal. This creates extra retries (false failures).
• Too long: stuck calls hold resources and cause cascading failures.

Also think about how long the user will wait. If the whole page must load in 1 second, one dependency cannot be allowed 5 seconds.

Deadline propagation

Deadline propagation means passing the remaining time along the chain. In a chain of services, each step (hop) should know how much time is left:

If the product service has only 200 ms left, it should not start a call that normally takes 400 ms. It should fail fast (give an error at once) or use a fallback (a simpler backup answer). gRPC has deadlines built in (post 31). In most languages you pass the incoming deadline on to the next call. With plain HTTP, you can send a deadline in a header.

Stop work when the caller has given up. If the user closed the page, or the service that called you (the upstream) has timed out, then going on with the work is a waste. Use cancellation (a way to tell running work to stop) where your framework supports it.

Part 2: Retries

A retry sends the same request again after a failure. Retries turn temporary failures into successes. But if you use them carelessly, they turn small problems into outages.

When to retry:

• ✅ Transient errors: timeouts, connection resets, 503 Service Unavailable, 429 Too Many Requests (after the Retry-After delay, a header in which the server says how long to wait), 502/504 from a proxy, and database deadlocks (two transactions that wait for each other).
• ❌ Permanent errors: 400, 401, 403, 404, 422, validation failures. Retrying will not help.

Only retry safe operations:

• Idempotent operations are safe to retry. Idempotent means that doing it many times has the same result as doing it once. GET, PUT and DELETE are idempotent.
• Non-idempotent operations (POST, "charge card") are safe to retry only with idempotency keys (post 34). An idempotency key is a unique label that lets the server spot a repeated request.

Retry amplification

Here is the hidden danger, called retry amplification: retries at many layers multiply. Imagine each layer makes 3 attempts (1 try + 2 retries):

With 4 attempts per layer across 5 layers, that's 4⁵ = 1,024 calls to the bottom service for one user action. When the bottom service is already struggling, this multiplication finishes it off.

Rules to prevent amplification:

1. Retry at one layer only, usually the one closest to the failure or to the edge. Do not retry at every layer.
2. Cap attempts, which means set a maximum. Usually 2–3 in total.
3. Use retry budgets. A retry budget is a limit on how many retries you allow. For example, retries may add at most 10% extra load. If more than 10% of recent requests are already retries, stop retrying and fail fast. (Envoy, gRPC retry throttling and some libraries support this.)
4. Honour Retry-After from servers. This means: wait at least as long as the server asks.
5. Combine with circuit breakers (post 42). A circuit breaker stops calls to a dependency that is clearly down, like a fuse that cuts the power.

Part 3: Exponential backoff

Exponential backoff means that the wait doubles after each failure. Do not retry at once. Wait longer after each failure:

wait = base × 2^(attempt − 1)       (attempt counts from 1; capped at some maximum)base = 100 ms:  attempt 1 → 100 ms  attempt 2 → 200 ms  attempt 3 → 400 ms  attempt 4 → 800 ms  attempt 5 → 1.6 s  … up to a cap (e.g., 10–30 s for background work)

This gives the struggling service time to recover, and it lowers the load while the service recovers.

Part 4: Jitter

Backoff alone has a flaw. Say 10,000 clients all fail at the same moment (for example, the database had a short problem). They all retry at exactly 100 ms, then 200 ms, then 400 ms. They stay in step and arrive in waves, and every wave hits the server at once. This is the thundering herd (a huge crowd of clients that all act at the same moment).

Jitter adds random time to the wait, so clients no longer act together. The most common forms:

(Here "attempt" counts from 0, so the first retry waits at most the base time.)Full jitter (recommended default):  wait = random(0, min(cap, base × 2^attempt))Equal jitter:  temp = min(cap, base × 2^attempt)  wait = temp/2 + random(0, temp/2)Decorrelated jitter:  wait = min(cap, random(base, previous_wait × 3))

AWS tested these variants and found that full jitter and decorrelated jitter spread the load much better than plain exponential backoff. They also finish the work with far fewer total calls.

async function withRetry(fn, { attempts = 3, base = 100, cap = 2000 } = {}) {  for (let i = 0; i < attempts; i++) {    try {      return await fn();    } catch (err) {      if (!isTransient(err) || i === attempts - 1) throw err;      const delay = Math.random() * Math.min(cap, base * 2 ** i);  // full jitter      await new Promise(r => setTimeout(r, delay));    }  }}

Jitter is not only for retries. Use it anywhere many clients act on a schedule:

• cron jobs (jobs that run on a time schedule) across thousands of servers,
• cache TTLs (the time after which a cached item expires; many items that expire together cause the cache avalanche in post 16),
• client reconnects after a WebSocket server restarts (a WebSocket is a long-lived connection, post 33),
• mobile apps that sync "every hour".

Putting it together: a checklist for every network call