From One User to Millions: Scaling a Web App on AWS

This is lesson 11, the last lesson of System Design Case Studies, Part 2. Earlier lessons went deep on single parts: the key-value store (Lesson 1) on partitioning and replicas, the CDN (Lesson 4) on edge caches, and multi-data-centre systems (Lesson 7) on what changes across regions. This lesson puts the stages in order, like the system-design-primer exercise solutions/system_design/scaling_aws ("Design a system that scales to millions of users on AWS"). For the build-it-yourself AWS steps, I link to the DevOps series at each stage instead of repeating them.

Words used in this lesson

• Vertical scaling means a bigger machine (more CPU, memory, disk). Horizontal scaling means more machines. See System Design Lesson 7.
• Stateless means a server keeps nothing between requests that another server would need. Then any server can answer any request.
• Bottleneck is the one resource that runs out first and limits everything else.
• Utilisation is how busy a resource is, as a fraction (0.8 = 80% busy).
• Latency is time for one request. Throughput is requests per second. p99 is the time that 99% of requests beat. See System Design Lesson 8.
• Read replica is a copy of the database that serves reads and follows the primary with a small delay. Replication lag is that delay.
• Cache hit ratio is the share of reads answered from the cache.
• Availability Zone (AZ) is one data centre (or group of them) in an AWS region, with separate power and network. A region is a geographic area with several AZs. See System Design Lesson 57.
• Failover is switching to a standby when the main one fails.
• Shard is one part of a split database. Each shard holds some of the rows.
• Little's law: the average number of requests in the system equals the arrival rate times the average time each spends in the system.
• SLO is a target such as "99% of requests under 300 ms". See System Design Lesson 47.

1. The rule: change only what a number tells you to

The primer's method is four steps, repeated:

1. Load test (send realistic traffic at a copy of the system).
2. Profile (find where the time or memory goes).
3. Fix the bottleneck, and compare options with their trade-offs.
4. Repeat.

Every stage below follows this. Each stage has a trigger (the number that moved), a change, and a cost (the new thing you must now run, secure, pay for and understand). Skipping stages is a common mistake: a team that starts with sharding and microservices for 200 users has paid every cost and gets none of the benefit. The opposite mistake is also real: waiting until the single database is on fire.

The numbers I use come from the primer: 10 million users, 1 billion writes a month, 100 billion reads a month, 1 KB per write. With 2.5 million seconds in a month that is about 400 writes per second and 40,000 reads per second on average, and 1 TB of new data per month (36 TB in three years). Peaks are higher.

2. The stages at a glance

A note on order: it is not strict. Multi-AZ (stage 9) is cheap insurance, and many teams do it at stage 3. A CDN (stage 5) is often the first and cheapest thing to add for a content site. Read the "trigger" column, not the stage number. The primer's own order is: single box, split DB and object store, load balancer and replication and CDN, cache and read replicas, autoscaling and automation, then queues, sharding and NoSQL.

3. Stage by stage

Stage 0. One box

One EC2 instance (a virtual server) runs the web server, the app and the database, plus an Elastic IP (a fixed public address) and Route 53 (DNS) pointing the domain at it. Open only ports 80, 443 and (from your own IP only) 22. This is where most products should start. A 2-user app does not need more.

• What breaks: the box is a single point of failure (System Design Lesson 40), and the database and web server fight for the same memory.
• Metric: CPU, memory, disk and IO from basic monitoring (CloudWatch or top). When memory or disk is steadily above about 70 to 80%, move on. The simplest first fix is vertical: pick a bigger instance. It is quick, but it gets expensive and it does not give redundancy.
• Learn it: DevOps Lesson 7 (EC2), DevOps Lesson 13 (Route 53), DevOps Lesson 2 (networking).

Stage 1. Split the database and the files

Move the database off the box to RDS (managed relational database: AWS does backups, patching, and optional standby). Move user files, images, JS and CSS to S3 (object storage, see System Design Lesson 17). Put the database in a private subnet so it has no public address, and only the app can reach it (DevOps Lesson 6 (VPC)).

• What breaks if you do not: database and web compete for resources; disk fills with uploads; you cannot size them separately.
• Metric: database memory and CPU on the same host as the web server; disk used percent; the share of requests that are for static files.
• Cost: each query now crosses the network (about a millisecond inside a region; measure yours). New security groups and IAM roles must be right (DevOps Lesson 5). RDS is a convenience you pay for compared with running the database yourself, but you also stop doing backups by hand.
• Learn it: DevOps Lesson 8 (RDS), DevOps Lesson 11 (S3 and CloudFront).

Stage 2. Add a cache

Put Redis (on ElastiCache, AWS's managed Redis and Memcached) next to the app. Cache hot, repeated reads (a profile page, a product, a feed) and move sessions (who is logged in) there, so servers do not keep them in memory. Read the cache first; on a miss, read the database and fill the cache ("cache-aside", System Design Lesson 16). The primer's advice is to first try the database's own cache settings and an index fix before adding a cache.

The size of the win depends on the hit ratio. With 40,000 reads per second and a cache in front (from my demo.ts):

hit 0%  -> db reads/s 40000hit 50% -> db reads/s 20000hit 90% -> db reads/s 4000hit 95% -> db reads/s 2000hit 99% -> db reads/s 400

Going from 90% to 99% cuts the database load by 10 times again. That is why cache hit ratio is the first number to watch.

• What breaks: stale data (the cache says the old name), a stampede (the cache is cold or a hot key expires, and thousands of requests hit the database together), and a cache outage that sends everything to the database.
• Metrics: hit ratio, evictions, cache latency, and database read QPS.
• Learn it: DevOps Lesson 16 (Redis), System Design Lesson 15 and Lesson 16.

Stage 3. Load balancer and stateless app servers

Run several app servers behind a load balancer (an ALB, Application Load Balancer, in AWS; see System Design Lesson 12). It spreads requests, checks health, and ends TLS (DevOps Lesson 4). This only works if the app is stateless: no sessions in local memory, no files written to local disk that later requests need. Put sessions in Redis, files in S3. Build the app as an image (DevOps Lesson 9 (Docker)) so every server is the same.

How many servers? Queueing maths gives a surprising answer. A single server with 50 ms of work per request, in the M/M/1 model (random arrivals, one worker):

util 0.50 -> 100.0 msutil 0.70 -> 166.7 msutil 0.80 -> 250.0 msutil 0.90 -> 500.0 msutil 0.95 -> 1000.0 msutil 0.99 -> 5000.0 ms

The 50 ms request takes 100 ms at 50% busy and 5 seconds at 99% busy. The formula is response = service time / (1 - utilisation). This is why you add servers before they are full, and why "CPU is 70%" is already a warning.

Now the number of servers. Say the site gets 2,000 requests per second, each needs 50 ms of server time, and we want the mean response at or under 75 ms. The work offered is 2000 x 0.05 = 100 busy servers. The M/M/c model (Erlang C) gives:

servers: 102  (offered load 100 busy servers, utilisation 98%)  101 servers -> mean 94.2 ms, P(wait) 88.3%  105 servers -> mean 55.2 ms, P(wait) 51.6%  110 servers -> mean 51.2 ms, P(wait) 23.7%  125 servers -> mean 50.0 ms, P(wait) 1.0%  150 servers -> mean 50.0 ms, P(wait) 0.0%lose 1 of 102: mean 94.2 ms

Two lessons. First, the model says 102 servers meets a mean of 75 ms, but at 98% utilisation almost every request waits, and one failure moves the mean to 94 ms. Means hide the tail (System Design Lesson 8 on p99). A sane plan is about 125 servers (80% utilisation), which leaves room for failures and bursts. Second, the real 50 ms is rarely constant, so real queues are worse than this model. Always load test.

• What breaks: a deploy or crash on the one server takes the site down; one slow instance drags responses; sticky sessions pile load onto one box.
• Metrics: app CPU and p95/p99 latency per instance, the LB's healthy-host count, 5xx rate, and requests in flight (Little's law: 1,000 requests per second at 0.2 s each means 200 requests in flight, from my demo, so a server pool needs room for that many concurrent requests).
• Learn it: DevOps Lesson 12 (ALB and Auto Scaling), DevOps Lesson 10 (Nginx), System Design Lesson 13.

Stage 4. Read replicas

With the primer's 100:1 ratio, reads dominate. Add one or more read replicas of the RDS database. Send writes to the primary and read queries to replicas (the app, or a proxy, decides). Replicas are copies that apply the primary's changes a little later (System Design Lesson 24).

After a 90% cache hit ratio the database sees 4,000 reads per second (from the table above). Spread over replicas:

1 replicas -> 4000 reads/s each2 replicas -> 2000 reads/s each4 replicas -> 1000 reads/s each8 replicas -> 500 reads/s each

• What breaks: replication lag. A user saves a profile, the next page reads from a replica that has not caught up, and the old value shows ("read your own writes" fails). Fixes: read from the primary for a short time after a write, or for the user's own data. Also, replicas do not help writes: every write still goes to the primary.
• Metrics: replica lag (seconds), primary CPU and write IOPS, per-replica read load, and connection counts.
• Cost: query routing in the app, lag handling, and more instances to run and pay for.
• Learn it: DevOps Lesson 8 (RDS), System Design Lesson 28 (consistency models).

Stage 5. CDN

Put CloudFront (AWS's CDN) in front of S3 and the site. Static files (JS, CSS, images, video) are served from caches near the user, and your servers never see most of those requests. Some pages and API answers can be cached for seconds or minutes too. Read Lesson 4 (the CDN case study) for how hit ratio, cache keys and purges work, and System Design Lesson 14 for the basics.

• What breaks without it: bandwidth and CPU spent on files; far-away users see slow loads.
• Metrics: CDN hit ratio, origin request rate, bytes served from origin, and time to first byte by region.
• Cost: stale files after a deploy (use hashed file names), cache-key mistakes (a cookie in the key kills the hit ratio), and purge delays.
• Learn it: DevOps Lesson 11 (S3 and CloudFront).

Stage 6. Queues and workers

Anything slow or bursty should leave the request path. The app puts a job on a queue (SQS in AWS) and answers at once; workers (EC2 or Lambda) take jobs off the queue. The primer's example: a photo upload goes to the app, the app queues a "make thumbnail" job, and a worker makes the thumbnail, updates the database, and stores it in S3. See System Design Lesson 18 (queues) and Lesson 38 (background jobs, retries, dead-letter queues).

A queue absorbs a burst, then drains. I ran a burst of 3,000 jobs per second for 60 seconds with 10 workers that each do 100 jobs per second:

capacity 1000/s, backlog after burst 120000, drain time after burst (no new arrivals) 120 swith 40 workers: capacity 4000/s, backlog 0

With 10 workers the queue grows by 2,000 jobs every second for a minute, which leaves 120,000 jobs waiting, and takes another 120 seconds to clear even if no new work arrives. Whether that is fine depends on the job: a thumbnail two minutes late is fine, a password reset email is not. With 40 workers there is no backlog. The right answer is a decision about how late is acceptable, then enough workers for that.

• What breaks: the queue grows without limit; a "poison" job fails forever; duplicates (queues deliver at least once, so workers must be idempotent; see Lesson 10 and System Design Lesson 37).
• Metrics: queue depth, age of the oldest message (the best alarm), worker error rate, and dead-letter queue size.
• Cost: work is now eventually done, not done on return. You need retries, a dead-letter queue, and a user interface for "processing".

Stage 7. Autoscaling

An Auto Scaling group keeps a number of servers between a minimum and a maximum, and adds or removes servers when a metric moves (CPU, request count per server, or queue depth for workers). The primer's reason: traffic that peaks in business hours, so you stop paying for idle servers at night and keep up with spikes.

• What breaks: scale-up takes minutes (boot, pull the image, pass health checks), so a sudden spike still hurts; the group can flap up and down if thresholds are too tight; a wrong max can bankrupt you, a wrong min can leave you thin.
• Metrics: the scaling metric itself, instance launch time, healthy count against desired count, and the cost trend.
• Cost: you now need fast, repeatable instance builds and good health checks. Do it with infrastructure as code so the setup is not by hand (DevOps Lesson 15 (Terraform)) and ship by pipeline (DevOps Lesson 14 (CI/CD)).
• Learn it: DevOps Lesson 12, System Design Lesson 58 (deployment strategies).

Stage 8. Shard, split, or move data

When the primary database's writes or size pass what one machine can do with headroom, replicas and caches no longer help (they handle reads). The primer lists the options: federation (split by function: users in one database, orders in another), sharding (split one table's rows by a key), denormalization, SQL tuning, and moving suitable data to NoSQL such as DynamoDB, or old data to a warehouse such as Redshift.

An example with the primer's numbers: 400 writes per second on average is easy, but peaks might be 10 times that. If one shard comfortably takes 1,500 writes per second and I want 50% headroom, then 5,000 writes per second needs:

-- 7. shards: 5000 writes/s, one shard takes 1500 writes/s, keep 50% headroom7 shards

The 1,500 figure is my assumption; measure your own database. See System Design Lesson 25 (sharding), Lesson 26 (consistent hashing) and Lesson 1 of this series (key-value store). For per-category "rank" queries after sharding see Lesson 9.

• What breaks: one hot shard; queries that cross shards (joins, global counts); resharding when you add shards; unique IDs across shards (Lesson 6 (unique ID generator)).
• Metrics: primary write IOPS and CPU, replication lag during write peaks, table and index size against memory, p99 of the slowest queries, and per-shard load balance.
• Cost: the largest so far, and the hardest to undo. Before doing it, try the cheap steps: indexes, better queries, archiving old rows, bigger instance, and batching writes.

Stage 9. Multi-AZ

An AZ can fail. Spread the app servers over at least two or three AZs (the ALB and the Auto Scaling group do this), and use RDS Multi-AZ, which keeps a standby copy in another AZ and switches to it if the primary fails. The primer puts multi-AZ in from the load balancer step on.

• What breaks if you do not: one data-centre problem is a full outage.
• Metrics: healthy hosts per AZ, failover time (test it), cross-AZ traffic.
• Cost: you run spare capacity so that losing one AZ still leaves enough servers. With 3 AZs and the need for N servers, each AZ must hold at least N/2 so any two can carry the load; that is 50% extra, with 2 AZs it is 100% extra. Also, cross-AZ data transfer may be billed. Check the vendor's pricing page.
• Learn it: System Design Lesson 40 (redundancy and failover), Lesson 45 (backups and disaster recovery), DevOps Lesson 18 (security and backups).

Stage 10. Multi-region

A second region gives lower latency for far users, survival of a whole-region failure, and a place to keep data that the law says must stay local. It is the biggest jump in complexity. The hard part is data: one region holds the writable copy (simple, but far users pay the latency and failover needs a promotion plan) or both regions accept writes (fast, but conflicts must be resolved). See Lesson 7 (multi-data-centre systems), System Design Lesson 27 (CAP and PACELC) and Lesson 28. Global DNS routing (Route 53 policies) sends users to the nearest healthy region (DevOps Lesson 13).

• What breaks: stale or conflicting data across regions; a failover that nobody has ever tested; config drift between regions.
• Metrics: cross-region replication lag, recovery time and data-loss window measured in a drill (RPO and RTO, System Design Lesson 45), and latency by user region.
• Cost: double the infrastructure and double the operations. Most products never need this. Many get most of the benefit from a CDN (stage 5) plus multi-AZ (stage 9).

4. How you know: metrics that decide each move

The thresholds are rules of thumb, not laws. Your SLO and your own load tests decide. For setting up the dashboards and alarms, see DevOps Lesson 17 (CloudWatch) and System Design Lessons 46 to 48.

5. The capacity demo

The numbers above come from capacity.ts (about 50 lines, shortened here):

// Little's law: requests in flight = arrival rate x time each spends in the systemexport const inFlight = (reqPerSec: number, secPerReq: number) => reqPerSec * secPerReq;// M/M/1: mean response = service / (1 - utilisation)export const mm1Response = (serviceMs: number, util: number) => (util >= 1 ? Infinity : serviceMs / (1 - util));// Erlang C: chance a request must wait with c servers and offered load a (in "busy servers")export function erlangC(c: number, a: number): number {  if (a >= c) return 1;  let term = 1, sum = 1;  for (let k = 1; k < c; k++) { term *= a / k; sum += term; }  const last = (term * a / c) * (c / (c - a));  return last / (sum + last);}export const mmcWait = (c: number, lambdaPerMs: number, serviceMs: number) => {  const a = lambdaPerMs * serviceMs;  return erlangC(c, a) * serviceMs / (c - a);};export const dbReadsAfterCache = (reads: number, hit: number) => reads * (1 - hit);export const readsPerReplica = (reads: number, replicas: number) => reads / replicas;export const shardsFor = (writes: number, perShard: number, headroom = 0.5) => Math.ceil(writes / (perShard * headroom));

demo.ts also starts from the primer's monthly totals:

-- 0. traffic from the primer's numbers (2.5M s per month)reads/s 40000  writes/s 400  ratio 100:1-- 2. Little's law: 1000 req/s x 0.2 s = in flight200 requests at once

Run node demo.ts next to capacity.ts. Both files use only the standard library.

6. Mapping to AWS, in plain words

The names change and new ones appear, but the roles stay. The point of the primer's exercise is the principles, not the product list.

7. Failure modes and trade-offs

Trade-offs to say aloud

• Simplicity vs headroom: every stage adds a part to run. Add a stage when a number says so, not before.
• Vertical vs horizontal: vertical is quick and capped; horizontal is not capped but needs stateless servers.
• Cache vs freshness: speed against stale data.
• Replicas vs consistency: read capacity against lag.
• Queue vs immediate answer: smooth load against "done later".
• Managed vs self-run: less work and less control. Managed services cost more per unit and save you people time.
• Multi-region vs everything else: do it last, and only if you must.

8. How to say it in an interview (60 seconds)

"I would start with one box and say why: it is cheap and fast to build. Then I scale by the bottleneck, and name the metric for each step. When the database competes with the web server for memory I move it to RDS and move files to S3. When identical reads dominate database CPU I add Redis; at 40,000 reads a second, a 90% hit ratio leaves the database 4,000. When one app server gets past about 70% CPU I add a load balancer and make the app stateless, with sessions in Redis. Queueing maths says latency explodes near full utilisation: a 50 ms request takes 500 ms at 90% busy, so I keep servers about 80% busy. Then read replicas for the 100-to-1 read ratio, accepting replica lag; a CDN for static files; SQS and workers for slow jobs, alarming on the age of the oldest message; autoscaling for the daily curve. Only when write rate or size passes one machine do I shard, and I try indexes and archiving first. Multi-AZ comes early for availability; multi-region last, because data consistency across regions is the hard part. At every step I load test, profile, and write down the cost I just took on."

Interview questions

Key takeaways

• Start simple (one box). Add a stage when a metric says the current one is running out: the trigger column is the useful part.
• Each stage fixes one bottleneck and adds one cost. Write the cost down.
• Stateless servers (sessions in Redis, files in S3) are what allow a load balancer and autoscaling.
• A cache fixes repeated reads, and hit ratio is the key number: 90% to 99% cut database reads from 4,000 to 400 per second in my example.
• Latency rises sharply near full utilisation: 50 ms became 500 ms at 90% busy and 5,000 ms at 99%. Plan for 70 to 80%.
• Replicas add read capacity and replication lag. They do nothing for writes.
• Queues smooth bursts. Alarm on the age of the oldest message.
• Shard last (after indexes, archiving and a bigger instance), and multi-region last of all.
• These formulas show the shape of the problem. Load test to get the numbers.

What's next

This is the last lesson of Part 2. To go back and fill gaps, the DevOps series shows how to build each AWS stage by hand, and System Design Lesson 59, a 5-step framework for any system design problem turns what you practised here into a repeatable interview method. If you want another pipeline-style case first, re-read Lesson 10 (the personal-finance data pipeline).