Design a CDN

This is lesson 4 of System Design Case Studies, Part 2. Lesson 1 (key-value store) showed consistent hashing and replicas inside one cluster. A CDN reuses these ideas at the world scale: thousands of caches, each holding a small slice of the content. For the simple "push or pull" overview of CDNs, see the system-design-primer and System Design Lesson 14, How CDNs Make the Web Fast if you want the basics first.

Words used in this lesson

• Origin is your own server (or object store) that holds the real content. A CDN copies from it.
• Edge (also PoP, point of presence) is a CDN site near users. It has cache servers. A big CDN has hundreds of PoPs.
• Cache hit / miss is when the edge has a fresh copy (hit) or has to go and get one (miss). Hit ratio is hits divided by all requests.
• TTL (time to live) is how long a copy is treated as fresh. After that it is stale.
• Cache key is the text that decides "is this the same object?". Two requests with the same key share one copy.
• LRU (least recently used) is an eviction rule: when the cache is full, throw away the entry that was used longest ago.
• Purge (invalidation) is telling the CDN to remove or stop using a copy before its TTL ends.
• Origin shield is one extra cache tier between all edges and the origin, so the origin sees one request instead of one per edge.
• Request coalescing (also called request collapsing) means: when many requests ask for the same missing object at the same time, send only one request to the origin and make the others wait for it.
• Thundering herd is the problem coalescing solves: an object expires, and thousands of requests all go to the origin at once.
• Anycast is one IP address announced from many places; the network (BGP, the routing system of the internet) sends each packet to the nearest one. GeoDNS is a DNS server that gives different answers to users in different places.
• TLS is the encryption used by HTTPS. TLS termination means the edge, not the origin, decrypts the connection.
• Zipf distribution is a popularity pattern where the object ranked k has a share proportional to 1/k^s. Web traffic often looks roughly like this (an assumption I use, not a law).

1. Requirements

Functional

• Serve static files (images, scripts, video segments, downloads) from a cache near the user.
• On a miss, fetch from the origin, store the copy, and serve it.
• Let the owner control freshness: TTL headers, a purge by URL, by prefix, or by tag.
• Support HTTPS with the customer's own domain and certificate.
• Optional: simple rules (redirect, header edit), compression, and signed URLs for private content.

Non-functional

• Low latency: the edge answers in tens of milliseconds for a hit.
• High hit ratio: the point is to keep traffic away from the origin. Target above 95% for static content (assume).
• Availability: if one PoP or one cache server dies, users still get content from another.
• Origin protection: a cold cache, a purge or a viral object must not flatten the origin.
• Freshness bounds: a purge should take effect everywhere in seconds (assume a target of a few seconds).
• Cost: bandwidth is the biggest bill; every miss costs origin egress.

Out of scope here: live video protocols, DDoS filtering details, and the billing system.

2. Back-of-envelope numbers

Assume a mid-size CDN customer base. Every number below is an assumption.

Peak traffic              100,000 requests/sAverage object size       500 KB (mix of images, scripts, video segments)Edge egress               100,000 x 500 KB x 8 bits = 400 Gbit/sPoPs                      200, each with ~20 cache servers -> 4,000 serversPer server (average)      400 Gbit/s / 4,000 = 0.1 Gbit/s    (peak is much higher in busy PoPs)Catalogue                 1 billion distinct objects x 500 KB = 500 TB at the originDisk per cache server     20 TB SSD -> 4,000 x 20 TB = 80 PB total, but each PoP holds only                          what its own users ask for (about 20 x 20 TB = 400 TB per PoP)

The key arithmetic is the hit ratio. I ran this small program (maths.ts):

hit ratio | origin req/s | origin load vs 90% | avg latency    50.0% |       50000 |               5.00x | 135.0 ms    90.0% |       10000 |               1.00x | 43.0 ms    95.0% |        5000 |               0.50x | 31.5 ms    99.0% |        1000 |               0.10x | 22.3 ms    99.9% |         100 |               0.01x | 20.2 msegress at the edge: 400 Gbit/sorigin egress at 95%: 20 Gbit/s

(Assumed: 20 ms for a hit, 250 ms for a miss.) Two lessons from the table:

1. Origin load is R x (1 - h). Going from 90% to 99% does not give "9% better": it gives 10 times less origin load. Think in miss ratio, not hit ratio.
2. Average latency is h x Lhit + (1 - h) x Lmiss. At 95% the 5% misses still make the average 31.5 ms against 20 ms for a pure hit. The tail (the slow 1%) matters more than the average, because a page loads many objects.

The system-design-primer's latency table gives the physics: a round trip inside one data centre is about 0.5 ms, and a California to Netherlands round trip is about 150 ms. A CDN turns the 150 ms trip into a trip to a city nearby. (Those are the primer's rounded numbers, not fresh measurements.)

3. API

A CDN has two APIs: the one users hit (plain HTTP, no change for them) and the control API the customer uses.

# Data plane (users): unchanged URLs, only the DNS name points at the CDNGET https://static.example.com/img/cat.jpg?w=200→ 200 OKCache-Control: public, max-age=86400, s-maxage=3600Age: 412                      # how long this copy has been in the cache (seconds)X-Cache: HIT                  # many CDNs add a header like this; the name differs by vendor# Control plane (customer)POST /v1/purge           { "urls": ["https://static.example.com/img/cat.jpg"] }POST /v1/purge           { "prefix": "https://static.example.com/img/" }POST /v1/purge           { "tag": "product-1234" }          # surrogate key / cache tagPUT  /v1/services/{id}   { "origin": "origin.example.com", "defaultTtl": 3600, "shield": "eu-west" }GET  /v1/stats?from=...  → hit ratio, bytes, origin requests, errors

s-maxage is a Cache-Control value that applies to shared caches (a CDN) while max-age applies to browsers too. Using both lets the CDN keep a file longer than browsers do (from memory; I did not re-read RFC 9111).

4. Data model

The edge does not use a relational database. Each cache server keeps:

index  (in memory):  cacheKeyHash -> { diskOffset, size, etag, lastModified, createdAt, expiresAt, tags[], varyKey }store  (on SSD):     object bytes + stored response headers at diskOffset

The control plane keeps ordinary configuration in a replicated database:

service(id, domain, origin_url, default_ttl, shield_region, cert_id, rules_version)purge_request(id, service_id, kind[url|prefix|tag], pattern, created_at, status)certificate(id, domains[], not_after, private_key_ref)

Config is pushed to all edges as versioned files and loaded in memory, so a request never calls the control plane.

5. High-level architecture

 User in Mumbai                                           Control plane      |  1. DNS: static.example.com  -> CNAME -> cdn-net.example (an IP near the user)      v                                                    (config, purge, certs, stats) +-----------+   2. TLS handshake + HTTP                         |  push config / purge fan-out |  Edge PoP |<-----------------------------------------------   v |  Mumbai   |   (anycast IP or GeoDNS picked this PoP)     [ all edges ] | [LB] [cache][cache][cache]   hash(cacheKey) picks one server in the PoP +-----------+      | miss      v +-----------+   shared by all edges in a region (fewer places that call the origin) |  Shield   |   one request per object per TTL reaches the origin |  Singapore| +-----------+      | miss      v +-----------+ |  Origin   |   your servers or object storage +-----------+

Request path for one GET:

1. The user's resolver asks DNS for the name. GeoDNS or anycast returns an edge near the user.
2. The edge ends TLS, builds the cache key, and picks the cache server in the PoP that owns this key (consistent hashing, as in Lesson 1). This way a PoP with 20 servers holds one copy of each object, not 20.
3. Fresh copy: send it (hit). Missing or stale: ask the shield. The shield does the same check, and only on a miss calls the origin.
4. Each tier coalesces: one fetch per key at a time, others wait.
5. The response is stored by every tier it passes (if the headers allow), then sent to the user.

6. Deep dives

6.1 Getting the user to a nearby edge

Two main tools. In real systems they are often combined.

A simple rule I use: DNS for coarse steering and control, anycast for fast failover and DDoS spread. I did not re-read any vendor's paper on this, so I state it as a common pattern and not as the way one named CDN does it.

Failover: health checks mark a PoP or server bad. With DNS you stop returning it (but old answers live for the DNS TTL, so keep the TTL short, such as 30 to 60 seconds; assumption). With anycast you withdraw the route.

6.2 TLS at the edge

HTTPS must end at the edge, or the edge cannot read the URL to find the cache key. Costs and tricks (from memory, standard practice):

• A TLS handshake needs round trips. If the edge is 10 ms away, the handshake takes tens of ms; if the origin is 200 ms away, hundreds. So the edge is a good place to end TLS. The edge keeps long-lived, reused connections to the origin or shield, so the user's handshake is short and the edge-to-origin leg pays the handshake once.
• Certificates: the customer either uploads one, or the CDN requests one automatically for the customer's domain (e.g., through the ACME protocol used by Let's Encrypt). Many certificates mean the edge needs to choose the right one by SNI (Server Name Indication: the name the client sends in the handshake).
• Private keys are the most sensitive thing at the edge. Keep them in memory or in a hardware module, never on shared disk in plain form.
• Session resumption lets a returning client skip most of the handshake.

6.3 The cache key

The cache key decides what counts as "the same object". Varnish's default hash, from its docs, is the URL plus the Host header (and it does not lowercase either, so Varnish.org/ and varnish.org/ are different entries in theory). Too narrow and users get the wrong content. Too wide and the cache fills with duplicates and the hit ratio falls.

Common rules:

• Lowercase the host. Sort the query parameters. Drop parameters that do not change the response (utm_*, fbclid).
• Add only the request headers that change the response (Vary). For example Accept-Encoding (gzip or brotli) or Accept-Language. Each added header multiplies the number of copies.
• Never put cookies in the key for public content. A cookie is nearly unique per user: it makes every request a miss.

This is the builder I ran (key.ts):

const DROP = /^(utm_|fbclid$|gclid$)/;export function cacheKey(rawUrl: string, headers: Record<string, string> = {}, vary: string[] = []): string {  const u = new URL(rawUrl);  const kept = [...u.searchParams.entries()].filter(([k]) => !DROP.test(k)).sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0));  const q = kept.map(([k, v]) => `${k}=${v}`).join("&");  const v = vary.map((h) => `${h}:${(headers[h] ?? "").toLowerCase()}`).join("|");  return `${u.hostname.toLowerCase()}${u.pathname}${q ? "?" + q : ""}${v ? " #" + v : ""}`;}

cdn.example.com/img/cat.jpg?h=100&w=200cdn.example.com/img/cat.jpg?h=100&w=200same key: true                              <- different order, different case, tracking params: one copycdn.example.com/app.js #accept-encoding:brcdn.example.com/app.js #accept-encoding:gzipsame key: false                             <- correct: brotli and gzip bytes differ

How much does it matter? In the simulation (test E in cdn.ts) I made 30% of requests carry a unique tracking parameter. With the parameter in the key the hit ratio was 46.3% and the origin got 536,658 requests. With the parameter removed from the key it was 73.4% and 265,542 origin requests: half the origin load, from one line of configuration.

6.4 TTL, freshness and revalidation

A copy is fresh until expiresAt = fetchedAt + TTL. The TTL comes from Cache-Control: s-maxage / max-age (or Expires), or from a default the customer sets. After it expires the edge has two choices:

• Fetch again (a full miss). Simple.
• Revalidate: send the origin a conditional request (If-None-Match: <etag>). If the file has not changed, the origin answers 304 Not Modified with no body, which is cheap. The edge resets the TTL.

Useful, widely supported extensions (from memory, not re-read in RFC 5861): stale-while-revalidate lets the cache serve the stale copy at once and refresh it in the background; stale-if-error lets it serve the stale copy when the origin fails. nginx has matching options (proxy_cache_use_stale, proxy_cache_background_update). Both turn an origin outage into "old but working" for users.

TTL choice is a hit-ratio choice. I ran the same traffic with different TTLs (100,000 objects, 1 million requests, cache big enough that size does not limit):

TTL    5 s  hit ratio  54.9%  origin requests 450917TTL   30 s  hit ratio  69.2%  origin requests 307967TTL  300 s  hit ratio  80.0%  origin requests 199941TTL 3600 s  hit ratio  80.7%  origin requests 192987

Short TTLs hurt a lot; after a few minutes the gain flattens out, because the remaining misses are objects that are requested once (the long tail), and no TTL helps them.

Best practice for static assets: put a content hash in the file name (app.3f9a1c.js) and give it max-age=31536000, immutable. The URL changes when the content changes, so you never need to purge it. Reserve purging for things that must keep their URL.

6.5 Invalidation and purge

"There are only two hard things..." the joke goes, and cache invalidation is one of them. The options:

How a purge reaches thousands of servers: the control plane writes the purge to a durable log, and each PoP reads it (or the control plane fans out through a tree: control plane -> regional hubs -> PoPs -> servers). Each server applies it and acknowledges. A new or restarted server must replay the log from its last position, or it could serve a copy that was purged while it was off. The purge status is shown as "complete" only when all PoPs acknowledged, or when a time limit has passed.

6.6 Thundering herd and request coalescing

When a hot object expires (or is purged), every request in the next fraction of a second is a miss. Without protection, each one goes to the origin. The nginx docs describe the fix for proxy_cache_lock: "only one request at a time will be allowed to populate a new cache element ... Other requests of the same cache element will either wait for a response to appear in the cache or the cache lock for this element to be released" with a timeout (default proxy_cache_lock_timeout 5 s; when it runs out, the request goes to the origin but the response is not cached). The Varnish docs say the same idea: it does coalescing automatically.

This is the core of my simulation (an excerpt of cdn.ts). A tier keeps one entry per key with a readyAt time. An entry that exists but is not ready yet is a fetch in progress:

request(key: string, t: number): number {  const e = this.cache.get(key);  if (e && e.expires > t) {    this.cache.delete(key); this.cache.set(key, e);          // touch: most recently used goes last    if (e.readyAt <= t) { this.hits++; return t; }            // fresh and ready: hit    if (this.coalesce) { this.coalesced++; return e.readyAt; } // a fetch is running: wait for it  }  this.fetches++;                                             // miss: ask the parent (shield or origin)  const done = this.parent.request(key, t + this.rtt / 2) + this.rtt / 2;  if (!(e && e.expires > t && e.readyAt > t)) {    this.cache.delete(key);    this.cache.set(key, { readyAt: done, expires: done + this.ttl });  }  while (this.cache.size > this.capacity) this.cache.delete(this.cache.keys().next().value as string);  return done;}

(A JavaScript Map remembers insertion order, so the first key is the least recently used. That is the whole LRU.)

Test C: one hot object, 2,000 requests/s for 20 s, TTL 5 s, the origin takes 200 ms:

coalescing OFF  requests 40000  origin fetches 1603  coalesced waiters 0     slowest client wait 200 mscoalescing ON   requests 40000  origin fetches 4     coalesced waiters 1599  slowest client wait 200 ms

Four expiries in 20 s means four fetches with coalescing: 400 times fewer origin requests. Without it, each expiry lets about 400 requests through (2,000 per second x 0.2 s while the first fetch is running), which is 1,603 in total. Real herds are worse than this one: more edges, slower origins, and an origin that gets slower under load which makes the herd longer. Note that the client wait is the same (200 ms) in both: coalescing does not slow anyone down.

Related protections: (1) Stale-while-revalidate: waiters get the stale copy at once, so nobody waits. (2) TTL jitter: add a random +/- 10% to the TTL so a million objects fetched together do not expire together. (3) Negative caching: cache 404s and origin errors for a short time (5 to 30 s) so a broken link does not become a stream of origin requests.

6.7 Origin shield

With 200 PoPs and no shield, one popular object is fetched from the origin once per PoP every TTL. A shield tier fixes this: all edges in a region ask the shield first. Test D: 20 edge caches, each holding only 2% of the catalogue; one shield holding 20%:

shield OFF  edge hit ratio 56.8%  origin requests 432339  (43.2% of all requests)shield ON   edge hit ratio 56.8%  shield hit ratio 55.0%  origin requests 194660  (19.5% of all requests)

The edges are the same. The shield turns many small caches into one big one for the misses: origin requests fall from 43.2% to 19.5%, 2.2 times less. The cost is an extra hop on a double miss (edge -> shield -> origin), and the shield is a hot spot, so it needs its own capacity and its own coalescing. Choose the shield location near the origin or in a well-connected region.

6.8 How big must the cache be? (the LRU simulation)

Test A uses 100,000 objects, 1 million requests, a Zipf popularity with exponent s (higher s means the traffic is more concentrated on a few objects), TTL one hour. "Ideal" is a cache that always holds the most popular C objects (no real cache knows the future, so it is an upper bound):

   zipf s | cache size       | simulated LRU | ideal 'top-C' | origin req   0.8    |   1000 ( 1%)   |         20.5% |         34.0% | 795067   0.8    |   5000 ( 5%)   |         36.9% |         50.5% | 630719   0.8    |  10000 (10%)   |         46.7% |         59.5% | 532823   1.0    |   1000 ( 1%)   |         50.7% |         61.9% | 493268   1.0    |   5000 ( 5%)   |         66.5% |         75.2% | 335226   1.0    |  10000 (10%)   |         73.5% |         81.0% | 264591   1.2    |   1000 ( 1%)   |         79.6% |         85.2% | 204467   1.2    |   5000 ( 5%)   |         88.5% |         91.9% | 115416   1.2    |  10000 (10%)   |         91.4% |         94.3% | 85628

(A "hit" here includes a request that waited for a fetch already running.) What I take from it:

• Skew decides everything. The same 1% cache gives 20.5%, 50.7% or 79.6% depending on how concentrated the traffic is. So the first real task is to measure the traffic, not to guess the cache size.
• LRU loses 3 to 14 points to the ideal in this test, because one-off requests push useful entries out. Fixes used in practice: admission rules (do not store an object until it has been requested twice, often done with a Bloom filter that remembers "seen once"), or segmented LRU. I did not test those here.
• Diminishing returns: for s = 1.0, going from 5% to 10% of the catalogue in the cache adds 7 points, from 1% to 5% adds 16 points.
• A mix of tiers helps: small hot objects in memory, big cold objects on SSD.

6.9 Where do 20 servers in a PoP fit?

If a PoP has 20 servers and each request goes to a random one, each server caches the same hot objects, and the PoP's total capacity is one server's capacity. If you instead map hash(cacheKey) to a server (consistent hashing, see Lesson 1), the PoP holds 20 different sets of objects. The trade-off is a very hot object, which hits one server only. Fix: replicate a hot object on a few servers, or put a small in-memory cache in the front-end tier.

7. Failure modes and trade-offs

Trade-offs to say aloud

• Freshness vs hit ratio: a long TTL means higher hit ratio and staler content. Versioned URLs let you have both.
• Push vs pull: pull (fetch on first request) is simple and stores only what is used. Push (upload in advance) suits big releases where you cannot afford a cold start.
• Shield: less origin load, one more hop, one more thing to run.
• Anycast vs DNS: fast failover and simplicity against fine control.
• Purge speed vs cost: faster global purge needs a more complex fan-out.

8. How to say it in an interview (60 seconds)

"A CDN is a set of caches near users. The user reaches the nearest PoP through GeoDNS or anycast; the edge ends TLS, builds a cache key, and uses consistent hashing to pick the server in the PoP that owns the key. A hit is served from there. A miss goes to a regional origin shield, and only on a double miss to the origin. Every tier coalesces requests for the same key, so a hot object that expires costs one origin fetch, not thousands; in my simulation that was 4 fetches instead of 1,603. The cache key must drop tracking parameters; in my test that raised the hit ratio from 46% to 73%. Freshness comes from TTL plus stale-while-revalidate, and I prefer content-hashed URLs so I rarely purge; when I must purge, I use tags and a durable log that every server replays. The number I watch is the miss ratio, because the origin load is request rate times miss ratio: going from 90% to 99% hits means ten times less origin load."

Run it yourself

The three files are cdn.ts (about 55 seconds, it runs 1 million requests many times), maths.ts and key.ts. Run each with node file.ts. They use only Node's standard library and Node 22's built-in type stripping (no constructor parameter properties, so I assign fields by hand).

Interview questions

Key takeaways

• A CDN is a tree of caches: edge -> shield -> origin. Users reach the edge by GeoDNS or anycast. The edge ends TLS.
• The cache key is the main hit-ratio control. In my test, a tracking parameter in the key cost 27 points of hit ratio.
• Think in miss ratio: origin load = request rate x (1 - hit ratio). 90% to 99% is 10 times less.
• Coalesce requests at every tier. Measured: 1,603 origin fetches became 4 for one hot object, with no extra wait.
• A shield cut origin requests from 43.2% to 19.5% in the test.
• Prefer content-hashed URLs; purge by tag with a durable log; use soft purge and stale-while-revalidate to avoid spikes.
• All simulation numbers are from made-up traffic. Measure your real traffic.

What's next

Lesson 5: Design Trending Topics and Top-K Requests. A CDN sees billions of requests and wants to know "which objects are hottest right now" without storing a counter for every object. That needs counting with small, approximate memory.