Design a System Across Multiple Data Centres

This is lesson 7 of System Design Case Studies, Part 2. Lesson 1 built a key-value store with replicas inside one cluster; this lesson stretches replication over a long, slow, sometimes broken link. Lesson 4 (CDN) already used DNS steering and caches near users. For the basics, see System Design Lesson 26 (replication), Lesson 27 (CAP and PACELC) and Lesson 45 (backups and disaster recovery: RPO vs RTO). The next case, Lesson 8 (stock exchange matching engine), shows a system that refuses to be active-active for a good reason.

Words used in this lesson

• Data centre (DC) is a building (or a cloud region, which is a group of them) with servers, power and network. An availability zone is one failure-isolated part of a region.
• Active-passive means one DC serves all traffic and the other waits as a standby. Active-active means both serve traffic at the same time.
• Failover is switching traffic from a failed site to a healthy one. Failback is moving back after the first site is repaired.
• RPO (recovery point objective) is how much data you can lose, measured in time ("up to 5 seconds of writes"). RTO (recovery time objective) is how long you can be down.
• Replication is copying data to another machine. Synchronous replication waits for the copy before saying "done". Asynchronous replication says "done" first and copies later.
• Replication lag is the delay between a write on the primary and the same write on the replica.
• DNS (domain name system) turns names into IP addresses. TTL (time to live) is how long a resolver may cache an answer. GSLB (global server load balancing) is a DNS service that answers differently depending on health and location.
• Split brain is when two sites each think they are the primary and both accept writes.
• Quorum is a majority vote. With 3 voters, 2 is a quorum.
• Fencing token is a number that grows with each new primary, so a storage system can reject writes from an old primary.
• Conflict is two different updates to the same data on two sites that did not know about each other.
• LWW (last write wins) is a rule: keep the value with the latest timestamp.
• Data residency is a legal or contract rule that some data must stay in a given country or region.

1. Requirements

Functional

• The service keeps working for users when one whole data centre is lost (power, network, fire, a bad deployment).
• Users are sent to a nearby, healthy site.
• Writes are not lost, or the loss is bounded and known (the RPO).
• After the failed site comes back, it rejoins without corrupting data.
• Some data (for example EU users' personal data) stays in its region.

Non-functional

• RTO: back in service in minutes, not hours (assume a target of 5 minutes).
• RPO: depends on the data. Payments: zero. A "like" counter: a few seconds is fine. Decide it per data type, not once.
• Latency: reads and writes should feel local to the user, as far as the laws of physics allow.
• Consistency: say which operations need a single truth (money, unique names) and which can be eventually consistent (System Design Lesson 28).
• Operability: failover must be a rehearsed, mostly automatic procedure, not an invention at 3 a.m.

Out of scope here: building the database replication protocol itself, and multi-cloud portability.

2. Back-of-envelope numbers

Everything here is an assumption or simple physics. I ran maths.ts.

Distance sets the floor for latency. Light in fibre travels at about two thirds of its speed in vacuum, so roughly 200,000 km/s (memory). I used great-circle distances (rounded, assumed):

route                                min RTT (straight fibre)same metro (50 km)                      0.5 ms   (real routes are longer; real RTT is higher)New York - London (~5,570 km)          55.7 msNew York - Singapore (~15,300 km)     153.1 msMumbai - Frankfurt (~6,600 km)         66.0 ms

You cannot configure this away. Real cables do not run straight, and routers add delay, so real round trips are higher. Use these as the best case.

What a synchronous remote write costs. Assume a local commit takes 2 ms and the write waits for one remote acknowledgement:

remote RTT   1 ms -> commit ~3 ms -> one client thread does at most 333 serial writes/sremote RTT  20 ms -> commit ~22 ms -> one client thread does at most 45 serial writes/sremote RTT  80 ms -> commit ~82 ms -> one client thread does at most 12 serial writes/sremote RTT 160 ms -> commit ~162 ms -> one client thread does at most 6 serial writes/s

Many threads in parallel get you more total throughput, but every user-visible write still waits 80 ms or more. This is the whole reason people use async replication across long distances.

Availability of two sites. If each DC is up 99.9% of the time and failures are independent:

one site 99.90% (525.6 min/yr down) -> two sites 99.9999% (0.5 min/yr down)

Treat this as an upper bound. Failures are not independent. A bad software release goes to both sites. A shared DNS provider or a shared configuration service fails for both. Most real multi-DC outages are caused by software and configuration, not by fire. Two sites remove the "building" risk and add the "coordination" risk.

DNS failover time (worst case, if resolvers obey the TTL):

check every 10s, 3 fails to declare down, TTL 300s -> detect 30s + TTL 300s = up to 330scheck every 10s, 3 fails to declare down, TTL 60s  -> detect 30s + TTL 60s  = up to 90scheck every 5s,  2 fails to declare down, TTL 30s  -> detect 10s + TTL 30s  = up to 40s

A 5-minute RTO is achievable with DNS only if the TTL is short. Some resolvers and clients cache longer than the TTL (memory), so keep the first number as a floor.

Data at risk during lag: writes per second x lag.

  1000 writes/s x 0.04 s lag = 40 writes at risk  1000 writes/s x 4 s lag = 4,000 writes at risk 50000 writes/s x 4 s lag = 200,000 writes at risk

3. API

Users see no change. The new API is for operators and for services that need to know where they are.

# Data plane: same URLs. Only the DNS name steers.GET https://api.example.com/v1/orders/42→ 200 OK   X-Served-By: eu-west-1     # many systems expose the serving site in a header, for debugging# Control plane (operators)GET  /v1/sites                     → [{ "site":"eu-west-1","role":"primary","health":"ok","replLagMs":38 }, ...]POST /v1/failover   { "to":"us-east-1", "mode":"planned|emergency", "fenceToken": 18 }POST /v1/failback   { "site":"eu-west-1", "after":"resync-complete" }PUT  /v1/traffic    { "weights": {"eu-west-1": 100, "us-east-1": 0} }    # drain, or shift gradually

Notes:

• mode: planned first stops writes on the old primary, waits for lag to be 0, and then promotes. Nothing is lost. emergency promotes without waiting and accepts the RPO.
• fenceToken increases on every promotion, so old primaries can be refused (section 6.7).
• Every write API that crosses sites should be idempotent (safe to repeat), because after a failover the client does not know whether its last request arrived. See System Design Lesson 34.

4. Data model

Decide for each kind of data where the truth lives:

User profile      home_region = "EU"   -> written only in EU, replicated read-only elsewhere (or not at all if residency says so)Orders            owner site per order (home region of the buyer)Counters / likes  CRDT counter, each site adds to its own slot; the total is the sumSession / cache   per site; can be lost; rebuilt on loginConfig / feature  one primary, replicated everywhere (rare writes)

Metadata each replicated row carries:

row_id, value, version / vector, last_modified_ts, origin_site, tombstone (for deletes)

And the control state, kept in a small store with a majority vote across three places (two DCs plus a witness, section 6.7):

cluster_state: { primary: "eu-west-1", epoch: 18, members: [...], updated_at }

epoch is the fencing token: a number that increases each time the primary changes.

5. High-level architecture

                          Users worldwide                                |                    GSLB / GeoDNS (health checks, TTL 30-60 s)                    /                            \        "eu users -> EU"                    "us users -> US"                   v                                 v     +-------- DC-EU (primary for EU data) ----+   +-------- DC-US (primary for US data) -------+     |  L7 load balancer                       |   |  L7 load balancer                         |     |  stateless app servers (N)              |   |  stateless app servers (N)                |     |  cache (per site)                       |   |  cache (per site)                         |     |  DB: EU shard leader + local replicas   |   |  DB: US shard leader + local replicas     |     |  queue / log                            |   |  queue / log                              |     +----------------+------------------------+   +----------------------+--------------------+                      |  async replication of changes (both directions)  |                      +<------------------------------------------------>+                                     ^                    ^                          cross-site link (80 ms RTT)     |                                                    Witness / tie-break (third site):                                                    holds the majority vote for "who is primary"

Reading the picture:

• Each site is self-contained: load balancer, app servers, cache, database, queue. A request is served entirely inside one site (no cross-site call on the hot path).
• Each piece of data has one home. Writes go to the home, then replicate to the other site asynchronously. This is "active-active at the service level, but single-writer per record".
• A third place (a witness) breaks ties, so that a network cut between the two DCs does not give two primaries.
• If DC-EU dies, GSLB sends EU users to DC-US, and DC-US promotes its replica of the EU data (section 6.6).

6. Deep dives

6.1 Active-passive vs active-active

My advice, which most real designs agree with: start with active-passive. It is much easier to reason about, because there is one truth. Move to "active-active with a home site per record" when you need low latency for far users or must use the standby's capacity. Go to true multi-writer only for data that merges safely, such as counters, sets and carts, and only on purpose.

6.2 Steering traffic: DNS, GSLB and anycast

How does a user reach the right site?

• GeoDNS / GSLB: the DNS server gives different answers by location (nearest site) and by health (never an unhealthy one). It runs health checks against each site. The cost is the TTL: after a site dies, resolvers may keep handing out its address until the cached answer expires. See the arithmetic in section 2: with 3 failed checks of 10 s and a 60 s TTL, the worst case is 90 s.
• Anycast: the same IP is announced from both sites and the internet's routing sends users to the closest. If a site stops announcing, traffic moves in the time routes take to change (often seconds to a minute, memory). No DNS cache problem, but you need network-level control. See Lesson 4.
• Client-side retry: mobile and web clients keep a list of endpoints and try the next one on failure. This beats DNS for failover time, and works only for clients you control.
• Weighted shift: use weights (95/5, then 50/50) to test the second site with real traffic before you need it.

Practical points: use a low TTL (30 to 60 s) only on the names that must fail over, and run DNS on a provider that is separate from your sites. Make health checks test something real (can the app read and write its database?), not just "port 443 answers", otherwise a site with a dead database stays "healthy".

6.3 Synchronous vs asynchronous replication, with numbers

This is the central trade-off. I wrote replication.ts to measure it in a simple model:

• The primary takes 1,000 writes per second (random arrivals) for 120 s.
• Each write is shipped over one link with a one-way delay of half the RTT (80 ms RTT assumed). The link can apply 3,000 writes per second normally.
• Async: the client gets its acknowledgement 1 ms after the write. Sync: the client gets the acknowledgement only after the secondary applied the write, plus the way back.
• I crash the primary at 2,000 random moments. A write is lost if it was acknowledged to the client but not yet applied on the secondary.
• Scenarios 2 and 3 add a congested link: during part of the run it can only apply 700 writes per second, so a backlog builds.

scenario                                              ack p50 lag p50 lag max  lost mean  lost p99  lost max  window maxasync, 80 ms RTT, link has 3x headroom                   1 ms   40 ms    42 ms       39.4        56        65       41 msasync, 80 ms RTT, 10 s congestion at 700 writes/s        1 ms   40 ms  4070 ms      304.0      3591      3939     4057 msasync, 80 ms RTT, 30 s congestion at 700 writes/s        1 ms   41 ms 12719 ms     2884.8     12361     12732    12678 mssync,  80 ms RTT, link has 3x headroom                  80 ms   40 ms    42 ms        0.0         0         0        0 ms

How to read it:

• Healthy async: the client sees 1 ms. The replica is about 40 ms behind. If the primary dies at a random moment, about 39 acknowledged writes are lost on average (up to 65). That matches the simple formula: 1,000 writes/s x 0.04 s = 40. The loss window ("window max") is at most 41 ms of data.
• Short congestion (10 s): lag reaches 4 seconds. A crash in the worst moment loses up to about 3,900 acknowledged writes, about 4 seconds of data. The mean is 304 because the crash is usually at a calm moment, but the worst case matters for your RPO.
• Long congestion (30 s): lag reaches 12.7 s and a crash loses up to 12,700 writes. The backlog keeps growing while the link is slower than the write rate.
• Sync: zero loss, but every write waits 80 ms (RTT).

So the RPO of an async system is not a constant. It is the lag at the moment of the crash, and lag is highest exactly when the network is in trouble, which is also when a failover is likely. Measure replication lag as a first-class metric and alert on it. Your real RPO is the worst lag you allow before you stop accepting writes or page someone.

Middle roads (memory, not tested here):

• Semi-synchronous: wait for the copy to reach one nearby replica (another availability zone, 1 to 2 ms away), and ship to the far DC asynchronously. You survive the loss of one machine or zone with no loss, and lose a small window only if the whole region goes.
• Quorum writes (3 sites, wait for 2): lose nothing if one site fails, pay the latency of the second-nearest site. Used by consensus-based databases (Raft or Paxos style).
• Synchronous only for the writes that need it: payments sync, "last seen" async.

6.4 RPO and RTO, set per data type

This table is what an interviewer wants to hear: "I would not use one rule for everything." A zero RPO costs latency and availability. Spend it only where the business needs it.

6.5 Conflicts when both sites can write

If the same record is written at both sites while they cannot talk (or before replication arrives), you have a conflict. The options, from simplest to most work:

1. Avoid them: one home site per record (section 4). Most systems should stop here.
2. Last write wins (LWW): keep the update with the later timestamp. Simple, and it silently drops the other update. It also trusts clocks that are never perfect.
3. Version vectors: each site keeps a counter per site for each record. You can then tell "A is newer than B" from "A and B are concurrent" and detect the conflict (memory). Detection is cheap; what you do next is the hard part: show both to the user, or merge in code.
4. CRDTs (conflict-free replicated data types): data structures whose merge always works, such as counters, sets, and registers (memory). Good for likes, carts, presence. Not for "bank balance must not go below zero", because no merge rule knows about that constraint.
5. Application merge: a function that knows the business (for example, a document merge). This is the area of Lesson 3 (collaborative editing).

I measured the cost of LWW in the same script. Both sites accept writes for 30 s during a partition (100 writes per second each, 200 keys), with site B's clock 50 ms fast:

== split brain: both data centres accept writes for 30 s, 100 writes/s each side, 200 keys ==writes accepted: 6000; keys changed on both sides: 200 of 200last-write-wins: A's value kept for 101 keys, B's for 99 keys; the other side's value is silently dropped for all 200a version-vector check would instead flag these 200 keys as conflicts for the app to merge

With a small key space, every key was changed on both sides, and LWW threw away one side's last update for every key. It also picked winners by the 50 ms clock difference, not by anything real. For a profile "bio" field that is acceptable. For an order status it is a bug. (The script keeps only the last write per key per side, so the real number of dropped updates is larger than 200.)

6.6 Failover and failback

Failover steps (emergency)

1. Detect: health checks and a human or automated decision. Beware false alarms: a flapping link can trigger a failover that causes more damage than the problem.
2. Fence the old primary: make sure it cannot accept writes any more (section 6.7). This comes before promotion.
3. Promote the replica in the other DC: it becomes the primary. Increase the epoch.
4. Check the data: you accept the RPO. Record what was lost: the last replicated position of the old primary versus the new primary's position.
5. Steer traffic: update GSLB, drain the old site.
6. Warm up: the new primary's caches are cold, which means a load spike on its database. Ramp traffic in steps.

Planned failover (a drill, or a migration) is easier: stop writes, wait for lag 0, promote, switch. RPO is zero. Practise it regularly. A standby that has never taken traffic is a belief, not a fact.

Failback is harder than failover, and people forget it:

1. The old primary comes back with a tail of writes that never reached the new primary. In my 10 s congestion scenario this tail was up to about 3,900 writes. These were acknowledged to users. You must decide: discard them (and tell users or support), or extract and re-apply them where they do not conflict. Never just let the old primary rejoin as primary.
2. Make the old primary a replica of the new one: wipe or roll back its extra tail, then resync from the new primary.
3. Wait for lag to reach about zero.
4. Optionally switch back with a planned failover, at a quiet time. Often you simply stay where you are, and the old site becomes the standby. There is no rule that the first site must be primary.

6.7 Split brain, quorum and fencing

Split brain: the link between the DCs breaks. Each site cannot tell "the other site died" from "the link is down". If both promote themselves, both accept writes, and the data diverges (section 6.5 shows the size of the damage).

Defences:

• A third voter (witness), in a third location. A site may be primary only if it can reach a majority of the three voters (memory: this is the idea behind Raft and Paxos style systems). With a cut link, the side that can still see the witness has 2 of 3 and stays primary. The other side has 1 of 3 and stops accepting writes. This is the CAP trade-off in action (System Design Lesson 27): in a partition, the minority chooses consistency over availability.
• Fencing tokens: each promotion gets a higher epoch number. The storage or the downstream service rejects writes carrying an older epoch. This protects against the zombie primary: a machine that was paused (a long garbage collection, a frozen virtual machine) and wakes up still believing it is primary.
• Lease with a timeout: the primary must renew its lease from the voters every few seconds. If it cannot renew, it stops. The new primary waits for longer than one lease before taking over. This depends on clocks running at about the same speed (not on them showing the same time), and it is a classic place for bugs.
• STONITH ("shoot the other node in the head"): power off or isolate the old primary through an out-of-band path before promoting. Blunt, but sure.

With only two sites and no witness you must choose: automatic failover with split-brain risk, or manual failover with a human as the tie-break. Many teams choose manual for the data layer and automatic for the stateless layer.

6.8 Latency in an active-active design

Cross-site calls are slow (section 2). Rules that follow:

• Keep a request inside one site. Do not make the hot path call another site. If a service must, use a timeout much shorter than the usual and a fallback.
• Read local, write home. Reads come from the local replica (maybe a little stale), writes go to the record's home site. If the user is far from the home site, the write is slower, so choose a home near where the user usually is.
• Read-your-own-writes. After a write the user must see it. A common trick: for a short time after a write, route this user's reads to the home site, or send a "min version" token with the next read, and make the replica wait until it has caught up to that version.
• Travelling users: a user from the EU who logs in from the US is routed to DC-US, whose copy of their data is read-only and a bit stale. Say what happens: redirect writes to the EU home, or serve slower.
• Queues are a friend: use a local queue and process slowly in the background, rather than a cross-site synchronous call.

6.9 Data residency

Some rules (company policy or national law) say that certain data must be stored and processed in a region. This changes the design:

• Tag data with a residency class at creation (home_region), and replicate only inside the allowed set of regions. Backups, logs, search indexes and analytics copies count too. This is where most violations happen.
• The "other site" for failover of EU data must itself be in the allowed area. With two EU regions that works. With one EU and one US region it does not, for personal data.
• Routing: a user's profile lives in their home region. Requests from elsewhere are proxied there or served with non-personal data only.
• Keys: if the rules allow only encrypted copies elsewhere, keep the encryption keys in the home region.
• Deletion must reach every copy in the region, including replicas and backups.

I am not a lawyer, and I did not check any specific law here. The design lesson is: residency is a data placement constraint that you have to put into the design at the start. Adding it later means moving data.

7. Failure modes and trade-offs

Trade-offs to say aloud

• Active-passive vs active-active: simplicity and one truth against capacity use and local latency.
• Sync vs async: zero loss and an RTT per write, against fast writes and an RPO equal to the lag.
• Automatic vs manual failover: fast recovery with split-brain and flapping risk, against a human delay.
• Consistency vs availability in a partition: the minority stops (consistent) or both continue (available, conflicting).
• Short DNS TTL vs resolver load and cache effects: faster failover against more DNS queries and some users ignoring it anyway.
• Cost: a second site, a witness, network links and extra people-time for drills. Do the maths against the cost of an outage.

8. How to say it in an interview (60 seconds)

"I start with RPO and RTO per data type, because that decides everything. For most data I would use active-passive with a warm standby, or active-active where each record has one home site, so there are no write conflicts. Users reach the nearest healthy site through GeoDNS or anycast with health checks that test the database, and a low TTL, so DNS failover takes about detection time plus TTL, say 90 seconds in the worst case. Replication across regions is asynchronous, because a synchronous write pays the round trip, 80 ms or more. My simulation showed async loses about write rate times lag: about 40 writes at 40 ms lag, but up to 4,000 when a 10-second congestion pushed lag to 4 seconds, so I alert on lag and use sync or quorum only for money. To avoid split brain I put a witness in a third place and use fencing epochs, so the minority side stops writing and a zombie primary is rejected. On failover I fence first, then promote, then steer traffic and warm up. On failback I resync the old primary as a replica and deal with its unreplicated tail. And for residency I tag data with a home region and restrict where copies, backups and logs go."

Run it yourself

Files: maths.ts and replication.ts. Run node maths.ts and node replication.ts in one folder. They use only Node's standard library and the numbers are repeatable because the random generator has a fixed seed (42). The simulation runs in about a second. Change ARRIVAL, the link capacity, the congestion window or rttMs in replication.ts and watch the "lost" columns.

Try it

Exercise 1. Your primary takes 20,000 writes/s. Replication lag is normally 50 ms and sometimes reaches 3 s. How many acknowledged writes can a crash lose in the normal case and the bad case? Answer: 20,000 x 0.05 = 1,000 writes normally. 20,000 x 3 = 60,000 in the bad case. The RPO you should quote is the bad case, or you must limit the lag.

Exercise 2. Two sites with a 70 ms RTT. A request does 4 database writes in order, each synchronous across sites. What is the minimum extra latency per request? Answer: 4 x 70 ms = 280 ms more than the local commit time. Fixes: batch the writes into one transaction (1 round trip), or replicate asynchronously.

Exercise 3. The link between your two data centres breaks, and no witness exists. Both sites still have users. What are your options? Answer: (a) Pre-decide a preferred primary site. The other site becomes read-only when it cannot see the preferred one. This is safe, but if the preferred site really died, the other stays read-only until a human promotes it. (b) Let both write and accept conflicts, then merge later (section 6.5 shows the cost). (c) Stop writes on both until a human decides. A witness in a third place makes the choice automatic and safe.

Interview questions

Key takeaways

• Decide RPO and RTO per data type first. Payments and a like counter do not need the same plan.
• Prefer one truth: active-passive, or active-active with one home site per record. True multi-writer is for data that merges.
• Distance sets the floor: New York to London is about 56 ms round trip in straight fibre. A synchronous cross-site write pays that on every write.
• Async replication loses about write rate x lag. In my model: about 39 writes at 40 ms lag, up to about 3,900 after a 10-second congestion, about 12,700 after 30 s. Alert on lag.
• DNS failover is detection time plus TTL (90 s in the example), and some clients ignore the TTL.
• Split brain is solved by a majority vote with a third voter, plus fencing epochs and leases. A minority side must stop writing.
• LWW silently drops updates (200 of 200 contested keys in my test). Use home sites, version vectors, CRDTs or an application merge.
• Failback is harder than failover: resync the old primary as a replica and handle its unreplicated tail.
• Practise failover. Each site should carry 100% of the load. Residency is a placement rule decided at the start.

What's next

Lesson 8: Design a Stock Exchange Matching Engine. It is the opposite case: one strict order of events matters more than being active in many places. We will see why a matching engine runs one sequencer per market, how it replicates its log, and how it fails over without losing an accepted order.