Graceful Degradation & Load Shedding

Part 1: Graceful degradation

Decide which features are critical and which are nice-to-have. Then make the nice-to-have ones optional when the system is under stress.

Example for an e-commerce site:

Degradation techniques:

• Serve stale or cached data. Stale means a little old. A cache is a store of ready-made answers. Show cached prices on product lists (while checkout checks the real price again), cached search results, or yesterday's "popular items" instead of live personal picks (posts 15–16).
• Simplify responses. Return fewer items per page, fewer fields and smaller images.
• Turn off expensive features with feature flags or kill switches. A feature flag is a setting that turns a feature on or off without a new release. A kill switch is a flag made to turn something off fast. You can flip them by hand, or automatically when load or errors go above a limit.
• Use static fallbacks. Show a pre-built page that a CDN serves. (A CDN is a network of servers that keeps copies of your pages near users.) For example, a ready-made homepage or product page.
• Go read-only. If the database cannot take writes, let users browse but show "Ordering is paused for a moment" instead of failing at random.
• Queue non-urgent work. Emails, analytics and notifications wait in queues (lines of jobs) and catch up later (post 18).
• Use circuit-breaker fallbacks (post 42) for dependencies that fail. A dependency is another service that you call.

Plan for degradation from the start. When you build each feature, ask:

• "What happens if this dependency is slow or down?"
• "Can this page render without it?"
• "Is there a flag to turn it off?"

A page that needs 12 services to all succeed is fragile. A page that skips 8 optional ones when they fail is resilient (it keeps working when parts fail).

Part 2: Load shedding

Load shedding means rejecting some requests on purpose when a server is overloaded, so it can serve the rest well. To shed is to drop. It is like a ship that throws some cargo overboard to stay afloat.

Why reject, and not just queue? Suppose a server accepts more work than it can do:

• requests queue up and latency (the time to answer) climbs (post 8),
• clients time out (give up waiting) after, say, 5 seconds, but the server keeps working on those requests anyway. It wastes effort on answers that nobody will receive,
• the timeouts trigger retries, which add even more load (post 41),
• memory fills up, garbage collection (the automatic memory clean-up) runs all the time, and the server may crash. That makes things worse for everyone.

Rejecting early and cheaply (in microseconds, with a quick 503 and Retry-After) keeps the server in its efficient zone. The requests that it accepts finish quickly.

This collapse is called congestive collapse: after some point, more load gives less useful work, not more.

When to shed

These signs tell you that a server is overloaded:

• Concurrency (in-flight requests, which means requests being handled right now) above a limit. This is often the most reliable sign.
• Queue length or queue wait time too high.
• CPU or memory above a threshold (a set limit).
• Latency rising above target.

Adaptive concurrency limits. Instead of a fixed number like "max 200 in-flight requests", some systems learn the right limit by themselves. This is similar to TCP congestion control (post 2), which is how the internet's main transport protocol slows down when the network is busy. The limit goes up while latency stays good, and goes down when latency climbs. Netflix's open-source concurrency-limits library and Envoy's adaptive concurrency filter work this way.

What to shed: prioritisation

Not all requests are equal. Label each request with a priority or criticality (how important it is), and shed the least important first:

• Requests that a user starts matter more than background ones (like a phone app that loads data early, called prefetching).
• Retries can be shed before first attempts. They are already part of the problem.
• Bots and crawlers go first.
• Paying customers or critical tenants (customer accounts) may get priority, but be careful to stay fair.

Where to shed

• At the edge or load balancer (the first machines that receive traffic): the cheapest place. Rejected requests never reach your servers.
• At the API gateway (the single entry point to your services): by route or priority (post 13).
• In each service: a small check at the very start of handling a request ("am I over my concurrency limit? Is this request low priority?").
• In queues: drop or delay low-priority jobs when the backlog (the waiting jobs) grows (post 38).

Smart queue handling under overload

• Bounded queues. A bounded queue has a maximum size. Never let a queue grow without a limit.
• Drop requests that have waited too long. If a request sat in a queue longer than the client's timeout, the client has already given up, so do not process it. (This is sometimes called deadline-aware or "controlled delay" queueing.)
• LIFO under overload. Normally a queue is FIFO (first in, first out, like a line at a shop). Some systems switch to LIFO (last in, first out, like a stack of plates) when overloaded. The newest requests are the ones whose clients are most likely still waiting. The old ones have probably timed out already.

Tell clients to back off

When shedding, respond with 503 Service Unavailable (or 429 for limits on one client) and a Retry-After header. Well-built clients (post 41) wait and retry with some random delay (jitter), and do not overload you.

Brownouts and waiting rooms

• Brownouts: a brownout is a partial power cut that dims the lights but does not turn them off. In software, you turn down non-essential features for a while on purpose (like image quality or recommendations) to reduce load, before things become critical.
• Virtual waiting rooms: use these for huge spikes that you can predict (concert ticket sales, product launches, big sales). Put users on a queue page ("You are number 18,402 in line") and let them into the site at a rate that the system can handle. This is load shedding with good manners.

Test it

As with failover (post 40), degradation that was never tested often fails when you need it:

• load-test (send fake heavy traffic) beyond your peak to see how the system behaves when overloaded,
• practise flipping kill switches,
• check that fallback pages and read-only modes really work.