Part 7 · Reliability & Resilience

Single Points of Failure, Redundancy & Failover

The site went down, and the culprit was none of the redundant boxes on the diagram but the one NAT gateway, the one DNS provider, or an expired certificate. How to hunt single points of failure, shrink blast radius, add redundancy that works, and fail over without split brain.

Lesson 4010 min readintermediate#reliability#availability

The Problem

Your architecture diagram looks solid:

a load balancer (a server that shares requests between many servers),
five app servers,
a Redis cache (a fast store for data that is read often),
a PostgreSQL primary (the main database) with a replica (a live copy of it).

Then one night, the whole site goes down. The cause is none of those boxes. It is the one NAT gateway that all outgoing traffic goes through. (A NAT gateway lets private servers reach the internet.) Or the one DNS provider (DNS turns a site name into an address). Or an expired certificate (the file that proves your site's identity for HTTPS). Or the one engineer who knew how to restart the payment connection, and who is on holiday.

In post 9 we learned how availability "adds up". In this part we learn how to build systems that keep working when parts break. Step one is to find every single point of failure (SPOF). Then you add redundancy and failover that really work. Redundancy means having a spare copy. Failover means switching to the spare when the main part fails.

The Core Idea

A single point of failure (SPOF) is any part that, when it fails, takes the whole system (or one critical feature) down. It is like one weak link in a chain.

Think about a hospital. It plans for failure:

It has backup generators in case the power grid fails.
Several doctors are on call (ready to come in), not just one.
There are two roads in, in case one is blocked.
Oxygen supply comes from a main tank and backup cylinders.
And they test the generators regularly. A backup that does not start when needed is worse than useless, because it gave you false trust.

Reliable systems follow the same rules: find the SPOFs, add a spare, make switching to the spare automatic, and test it regularly.

How It Works

Step 1: Find the SPOFs

Follow the path of a request (post 11) and ask about every box and arrow: "What happens if this dies right now?"

1
User
2
DNS
one provider? → nobody can find you
3
CDN · Load balancer
one instance? one zone? (A CDN keeps copies of your files near users.)
4
App servers
stateless (no user data kept on the server) and N+1 (one spare)?
5
Cache
can the database survive without it?
6
Database
automatic failover — and tested?
7
Payment · email · object storage
third parties need fallbacks or buffers

Follow the request path and ask about every box: what happens if this dies right now? · flow

Obvious SPOFs:

a single database primary (without automatic failover),
a single load balancer,
a single app server,
everything running in one availability zone (one separate data centre inside a cloud region) or one region (a cloud area made of several zones, such as US-East-1).

Hidden SPOFs, which cause many real outages:

DNS provider: one provider down means nobody can find you (post 1).
Certificates: one expired certificate breaks HTTPS for everyone (post 3).
NAT gateway, VPN or firewall: one box that all traffic must pass through. (A VPN is a private network link. A firewall blocks unwanted traffic.)
Configuration or feature-flag service: a feature flag is a setting that turns a feature on or off. If every service needs this service to start, it can block recovery.
Authentication service (the service that checks who a user is): if it is down, nobody can log in, even though everything else works.
A shared database or cache used by "independent" services.
Third-party APIs: payments, SMS, maps.
The deploy pipeline: you cannot release the fix if CI/CD is down. (CI/CD is the automatic system that tests and releases code.)
People and knowledge: "only Ravi knows how this works."
The monitoring system: if it runs on the same machines, it goes down together with the thing it should warn you about.

Step 2: Understand blast radius

Not every failure is equal. Blast radius is how much of the system (and how many users) one failure harms. Think of how far the damage of an explosion reaches.

Ways to make it smaller:

Isolate features. Keep them apart. If recommendations fail, checkout should keep working (post 44).
Cells. Split the system into several independent copies (cells). Each cell serves a part of your customers. A bad deploy or an overload in cell 3 harms only cell 3's customers.
Shuffle sharding. A shard is one part of a split system. Here, you give each customer a random combination of servers. Say one customer sends "poison" traffic (requests that crash servers) and kills its servers. Very few other customers share exactly the same combination, so they are barely affected.

Normal sharding (4 shards)

•A and B share shard 1
−A's poison traffic kills shard 1 → B is down too

Shuffle sharding (2 of 8 servers each)

•A → 1, 5 B → 2, 7 C → 1, 7
+A kills 1 and 5 → B unaffected, C loses only one server
+Few customers share exactly the same pair

Why shuffle sharding shrinks the blast radius of a bad customer. · compare

Step 3: Add redundancy

Redundancy means having more than one of anything critical, like a spare tyre in a car.

Redundancy at every layer:

Layer	Redundancy
DNS	Two DNS providers, or a highly redundant managed DNS
Edge	CDN (itself globally redundant), or even two CDNs
Load balancer	A pair that shares a floating IP (an address that moves to the healthy one), or a managed load balancer across zones
App servers	N+1 or more stateless servers across zones (post 7)
Cache	Replicas or a cluster; the app still works if the cache is lost (post 16)
Database	Primary + replicas in different zones with automatic failover (post 24)
Storage	Object storage replicated across zones (post 17)
Data centre	Multiple availability zones
Region	Multi-region for the most critical systems
Dependencies	Backup providers (a second SMS/email provider), and queues that hold work during an outage
People	Written runbooks (step-by-step guides), on-call rotations (a schedule of who is ready), shared knowledge

Capacity planning: N+1 and N+2. Capacity planning means working out how many servers you need.

N = the number of servers needed to handle peak load.
N+1 = one extra server, so you can lose one (or take one down to deploy) without overload.
N+2 = two extra servers, so you survive a failure during maintenance.

Zone maths. Say you run in 3 availability zones and must survive losing one. The remaining 2 zones must handle 100% of the traffic. So each zone needs capacity for 50% of peak (peak is the busiest moment). In total that is 150% of peak. With only 2 zones, each needs 100%, so you pay double (200%). This is why three zones is a common best choice.

Step 4: Failover, switching to the spare

Failover means moving traffic from a failed part to its healthy copy. Redundancy is only useful if traffic really moves to the healthy copy. There are two main designs.

Active-passive means one copy serves traffic and the other waits as a standby. A standby can be hot, warm or cold:

Active-passive

•Active takes all traffic; passive waits
•Hot (seconds) · warm (minutes) · cold (hours)
+Simple data story: one writer
−Idle hardware; failover must be detected and executed

Active-active

•Both serve traffic all the time
+No idle hardware; near-instant failover
−Each node needs headroom for the other's load
−Two writers means conflicts to resolve

Two ways to run a spare. · compare

Hot standby: running and up to date, so it takes over in seconds.
Warm standby: running, but smaller or a little behind. It takes minutes, and may need to be scaled up.
Cold standby: not running. It must be started and restored from backups, which takes hours.

Active-active means all copies serve traffic at the same time. The numbers below show how much capacity each choice needs:

2 zones

100% each

200% of peak in total

3 zones

50% each

150% of peak — the common sweet spot

N+1

one spare

survive a failure or a deploy

N+2

two spares

survive a failure during maintenance

Zone maths — capacity needed to survive losing one availability zone. · numbers

✅ No idle hardware, and failover is almost instant because both are already serving.
❌ For data, if both copies accept writes, conflicts can happen (multi-leader, post 24). Each node must have spare capacity (headroom) to take the other's load.

Failure detection: the hard part

How do you know that something has failed?

Health checks (a regular test that asks "are you working?") and heartbeats (a regular "I am alive" signal) (post 12).
Timeouts: for example, "no heartbeat for 10 seconds means dead".

The trade-off:

Detect too slowly: users see errors for a longer time.
Detect too eagerly: a short network problem or a long garbage-collection pause (a pause while the program cleans up memory) causes a failover that was not needed, and the failover itself disturbs users. The system can also flap, which means it switches back and forth again and again.

Split brain (post 24) happens when the "dead" node was really alive but cut off from the network. Now two nodes both think they are in charge. This can corrupt data. Prevent it with:

quorum: only the side that has a majority of nodes may act. (etcd, ZooKeeper and Raft are tools and protocols that use quorum),
fencing: make sure the old primary can no longer write. You can remove its access, power it off, or use fencing tokens (numbers that grow with each new leader, so storage rejects the old leader's writes).

Automatic vs manual failover

Automatic: fast (seconds), works at 3 AM, and needed for high availability goals. But a bad detector can start a failover that is not needed. Automation that triggers more automation can make an incident worse.
Manual: a person checks the situation first, which is safer for complex cases. But it is slow, and it needs someone who is awake and knows what to do.

Many teams automate common failovers that they understand well (promoting a database replica to primary inside one region). They keep human approval for big, risky ones (moving everything to another region).

Static stability

A statically stable system keeps working during a failure without having to make any change. For example:

pre-provision (prepare in advance) enough capacity in each zone. Then losing a zone does not require starting new servers. (The control plane, the system that starts servers, may be hurt by the failure too.)
keep a local copy of the configuration, so services keep running if the configuration service is down,
make sure recovery does not depend on the thing that failed.

Step 5: Test it, regularly

Failover that you never tested usually fails when you need it. Common ways to test:

Game days: planned practice days. The team breaks something on purpose (for example, fails over the database) and practises the response.
Chaos engineering: you add failures on purpose (kill servers, add delay, cut network links) in a controlled way, ideally in production, to prove that the system copes.
Disaster recovery drills: really restore from backups, and really fail over to another region (post 45).
Runbooks: written, step-by-step instructions for each kind of failure. Update them after every drill and incident.

Trade-offs

More redundancy: higher availability, but more cost (spare capacity, extra zones or regions) and more complexity.
Complexity is itself a risk. Failover systems can have bugs. Many big outages were caused or made worse by automation that was meant to protect the system.
Active-active: no idle capacity and instant failover, but it is harder to keep data consistent, and every node needs headroom.
Fast failure detection: quick recovery, but more false alarms (false positives) and flapping.
Multi-region: survives the failure of a whole region, but it is expensive, and it makes copying data and keeping it consistent much harder (posts 24, 27–28).

In the Real World

Netflix and Chaos Monkey. Netflix built Chaos Monkey, a tool that randomly shuts down production servers during working hours. This forces every team to build services that survive the loss of any server. Netflix later used the same idea for larger failures, including simulating the loss of an entire AWS region. Its "Simian Army" of tools helped make chaos engineering popular across the industry.

Big cloud region outages. Big cloud providers have had serious regional incidents. For example, AWS's US-East-1 region has had several outages that many people felt. One was in October 2025, when a DNS problem around DynamoDB spread to many other services. Companies that ran everything in a single region went down. Companies with multi-region failover or static stability stayed up or recovered faster. These events are the main reason large companies spend money on multi-region designs.

Certificate expiry outages. As in post 3, Microsoft Teams (2020) and O2's UK mobile network (2018, caused by Ericsson software) had outages because of expired certificates. A certificate is a classic hidden SPOF. The fix is automatic renewal plus monitoring of expiry dates.

Dyn DNS attack (2016). A massive attack hit the DNS provider Dyn. Many sites that relied only on Dyn could not be reached. Afterwards, many companies added a second DNS provider (post 1).

AWS cells and shuffle sharding. AWS has written about using cell-based architecture and shuffle sharding in its own services. Then a failure, or a customer who misbehaves, harms only a small part of the users and not everyone.

Interview Questions

Q1How do you find single points of failure?

Follow the full request path and every dependency, and ask “what if this dies now?” Include the hidden ones: DNS, certificates, NAT gateways, config and auth services, shared databases, third-party APIs, the deploy pipeline, monitoring on the same machines, and people who are the only ones with some knowledge.

Q2Active-passive vs active-active?

Active-passive keeps a standby (hot, warm or cold) that takes over on failure. It is simpler for data, but the standby sits idle and the failover must work. Active-active serves traffic on all nodes, so failover is almost instant and no hardware sits idle. But each node needs headroom, and data with many writers needs conflict handling.

Q3What is split brain, and how do you prevent it?

Two nodes both act as primary after the network splits (a partition). Prevent it with quorum (only the side with a majority may act, for example with Raft or etcd) and fencing (remove the old primary's access, power it off, or reject its old fencing tokens).

Q4What is static stability?

A system that keeps working through a failure without having to change anything. It has capacity prepared in every zone, a local copy of its configuration, and a recovery that does not depend on the control plane, which might be failing too.

Q5How much capacity do you need to survive losing one of three zones?

The two remaining zones must carry 100% of peak, so each zone needs 50% of peak. That is 150% of peak in total, compared with 200% for only two zones.

Q6How do you know your failover works?

You test it: scheduled game days, chaos experiments in production, real restores and region failovers, and runbooks updated after every drill. Redundancy that was never tested usually fails when it is needed.

Key Takeaways

A single point of failure is anything whose loss takes down the system. Walk the request path and hunt for hidden ones: DNS, certificates, NAT, config, auth, shared databases, third parties, people.
Shrink the blast radius with isolation, cells and shuffle sharding.
Add redundancy at every layer, and plan capacity (N+1, and zone math: 3 zones → 150% total capacity).
Failover can be active-passive (hot, warm, cold) or active-active. Detection is the hard part, so guard against false positives and split brain with quorum and fencing.
Aim for static stability, and test failover regularly (game days, chaos engineering, drills). Untested redundancy is only hope.

Keep learning

Part of the study guide

System Design Interview Guide: Basics to Case Studies

Related series