Part 2 · System Design Core Concepts
Vertical vs Horizontal Scaling (and Why Stateless Wins)
Bigger server or more servers? Vertical scaling is simple but hits a ceiling; horizontal scaling is nearly unlimited, but only once your app forgets everything between requests. Why stateless wins, and where state should live instead.
On this page
The Problem
Your app runs on one server, and traffic has doubled in three months. The CPU (the processor that does the work) is 90% busy, pages are getting slower, and a big sale starts next week. You have two options:
- Buy a bigger server.
- Add more servers.
Both work, but they lead to very different systems. And if you choose "more servers", you may get a surprise. Users keep getting logged out, and uploaded profile pictures randomly disappear. This is a state problem. It is the hidden reason why many apps cannot simply "add more servers". Scaling means making a system able to handle more load.
The Core Idea
Imagine a busy chai stall.
Vertical scaling (scale up) means giving your one chaiwala (tea seller) a bigger stove, a bigger pot and faster hands. It is simple. Nothing about how the stall works changes. But one person can only get so fast, bigger equipment gets very expensive, and if the chaiwala falls sick, the stall closes.
Horizontal scaling (scale out) means opening more counters with more chaiwalas, and having someone at the front send customers to whichever counter is free. You can keep adding counters almost forever, and if one chaiwala takes a break, the others keep serving. But now you need coordination. Say a customer paid at counter 1 and comes back to counter 3 for their change. Counter 3 needs to know about the payment.
The question "does counter 3 know about you?" is the stateless vs stateful idea. It decides how easily you can scale out.
How It Works
Vertical scaling
You move to a machine with more CPU cores, more RAM (the fast working memory), faster disks or a faster network. In the cloud this can be as easy as changing the instance size (the size of the rented virtual machine) and restarting.
Before: [ 4 CPU | 16 GB ]After: [ 64 CPU | 512 GB ]Good things:
- No code changes. Your app does not know that anything changed.
- No distributed-system problems. (A distributed system is a system that runs on many machines.) You have one machine, one memory and one clock.
- Good for databases, which are hard to split across machines.
Limits:
- There is a ceiling. Even the biggest cloud machines have limits.
- The cost grows fast. At the high end, a machine twice as big often costs more than twice as much.
- It is still a single point of failure. A single point of failure is one part that, if it breaks, stops the whole system. With one machine, one hardware failure takes you down.
- It usually needs a restart, which means downtime.
Horizontal scaling
You run many copies of your app on many machines. A load balancer sits in front of them. It is a server that spreads the incoming requests between the machines.
┌──► [ App server 1 ]
Users ──► [LB] ─┼──► [ App server 2 ] ──► Database
└──► [ App server 3 ]Good things:
- Almost unlimited growth. You just add servers.
- Fault tolerance. This means the system keeps working when a part fails. If server 2 dies, the load balancer sends traffic to servers 1 and 3.
- Cheaper hardware. You use many ordinary machines (commodity machines) instead of one huge one.
- Zero-downtime deploys. A deploy is putting a new version of your app on the servers. You update the servers one at a time, so the site never goes down.
Challenges:
- You need a load balancer (covered in Part 3).
- Your app must be stateless (see below).
- Some parts, especially the database, do not scale out easily. Part 4 covers replication (keeping copies of data on several machines) and sharding (splitting data across several machines).
Different tiers scale differently
- The web/app tier (the servers that run your application code) is usually easy to scale out, if it is stateless.
- Caches can be scaled out with partitioning (each server holds a different part of the data).
- Databases are hard. Most teams scale the database vertically first, then add read replicas (extra copies that only answer reads), and shard only when nothing else works.
A very common setup in real systems is many small app servers + one big database (plus replicas).
What is "state"?
State is any data that a server remembers between requests. Common examples:
- Login sessions kept in server memory. (A session is the server's record that you are logged in.)
- Uploaded files saved to the server's local disk.
- In-memory caches holding user-specific data.
- Open connections, like WebSockets for chat. (A WebSocket is a connection that stays open so that the server and the browser can send messages at any time.)
Here is why state breaks horizontal scaling:
- POST /login
- forward
- ↻ remember session 123 = Asha (in RAM)
- Set-Cookie: session=123
- GET /dashboard (cookie 123)
- forward — round robin (servers take turns)
- 401 “Please log in again” 😩B has never heard of session 123
The same thing happens with files. The profile picture was saved to Server A's disk, so when Server B handles the next request, the image "does not exist".
Making services stateless
A stateless server keeps nothing important in its own memory or disk between requests. Any server can handle any request. Difference in one line: a stateful server remembers you, a stateless server does not need to. You move the state to a shared place:
| State | Move it to |
|---|---|
| Login sessions | A shared store like Redis (a very fast in-memory data store), or signed tokens (like JWTs, small signed texts) that the client sends each time |
| Uploaded files | Object storage (a service that stores files, like Amazon S3) |
| Cached data | A shared cache (Redis / Memcached, a fast store of saved answers that all servers use) |
| Business data | The database |
Now servers are interchangeable, like identical counters at the chai stall. You can add servers, remove them, restart them or replace them at any time. This is the key idea behind the well-known "Twelve-Factor App" guidelines (a popular list of rules for building web apps): run your app as stateless processes.
Sticky sessions: a shortcut with costs
Another option is sticky sessions: the load balancer always sends the same user to the same server, usually using a cookie. It is a quick fix, but:
- Load becomes uneven. One server might get all the heavy users.
- When that server dies, its users lose their sessions.
- Scaling down or deploying is harder, because users are "attached" to specific machines. Scaling down means removing servers.
Use sticky sessions only when you really must, such as for some long-lived connections. Otherwise, make the app stateless.
Some things are naturally stateful
Not everything can be stateless. Databases, chat servers holding WebSocket connections, multiplayer game servers and stream processors all have to hold state. They are scaled with extra techniques:
- replication (copies),
- partitioning (each server owns part of the data or part of the users),
- routing that sends each user or key to the right server.
We cover these in later lessons.
Auto-scaling
Auto-scaling means the cloud adds and removes servers for you, based on a signal. It works well with stateless servers. Common signals:
- CPU usage (for example, "keep average CPU around 60%"),
- requests per second,
- queue length (for background workers).
Two practical points:
- New servers take time to start and warm up (to load code and fill caches), from seconds to minutes. Auto-scaling reacts after the load rises, so for spikes that you can predict (a sale, a big match), add servers in advance.
- Set a minimum number of servers, so that a failure never leaves you with none.
Why 10 servers aren't 10× faster
Adding servers rarely gives perfect linear growth (twice the servers, twice the speed):
- Shared bottlenecks. A bottleneck is the one slow part that limits everything else. All servers still use the same database, cache or third-party API (a service run by another company).
- Coordination costs. Servers may need to talk to each other, share locks (a lock lets only one worker use something at a time) or keep data in sync.
- Amdahl's law. If part of the work cannot be split, like one shared database write, that part limits your total speed-up, no matter how many servers you add.
Scalability is not just adding machines. It is removing the shared bottlenecks one by one.
- +No code changes at all
- +No distributed-systems problems
- −Hard ceiling on machine size
- −Price climbs steeply at the top end
- −Still one machine: a single point of failure
- +Nearly unlimited: just add servers
- +Survives a server dying
- +Zero-downtime rolling deploys (update servers one by one)
- −Needs a load balancer and stateless design
- −Shared bottlenecks (the database) remain
Trade-offs
| Vertical | Horizontal | |
|---|---|---|
| Simplicity | ✅ Very simple | ❌ Needs a load balancer (LB) and a stateless design |
| Max scale | ❌ Hardware ceiling | ✅ Nearly unlimited |
| Fault tolerance | ❌ Single point of failure | ✅ Survives server failures |
| Cost at large scale | ❌ Steep | ✅ Commodity machines |
| Best for | Databases, early-stage apps, quick fixes | Web/app tiers, workers, caches |
When vertical scaling is the right answer:
- Your app is small and a bigger machine solves the problem for years.
- The bottleneck is a database that is hard to split.
- You need a quick fix before a deadline, while you plan a longer-term change.
Do not scale out before making the app stateless. You will just get random logouts and missing files.
In the Real World
Stack Overflow is known for serving a huge global audience with a small number of powerful servers, including large SQL Server database machines. The team chose to scale up, and to tune performance carefully, instead of splitting into hundreds of services. This shows that vertical scaling can go a very long way.
Let's Encrypt, which issues certificates for hundreds of millions of websites, wrote about upgrading its main database to much more powerful servers instead of redesigning the database layer. This is a clear, practical example of choosing vertical scaling for a database.
Netflix runs its services as large groups of stateless instances in the cloud. It uses auto-scaling to add capacity in the evening when viewing peaks and to remove it overnight. Because the instances are interchangeable, Netflix can even stop random ones on purpose in production (with its "Chaos Monkey" tool) to prove that the system survives.
Live sports streaming platforms know exactly when the big match starts, so they pre-scale (add servers early) before the first ball instead of relying only on auto-scaling. Waiting for the CPU load to rise would be too late when millions of people join within minutes.
Your own projects. If you deploy a Node.js or Next.js app to a platform like Vercel, Render or Kubernetes, you are often running several instances without realising it. Keeping sessions in memory or files on local disk "works on my machine", then breaks in production for exactly the reasons above.
Interview Questions
Q1When would you scale vertically rather than horizontally?
When the app is small and a bigger machine gives you years of room to grow, when the bottleneck is a database that is hard to split, or as a quick fix before a deadline. It keeps the system simple. The price is a ceiling and a single point of failure.
Q2What does it mean for a service to be stateless, and why does it matter?
A stateless server keeps nothing important in its own memory or disk between requests. Sessions, files and caches live in shared stores. Then any server can handle any request, so you can add, remove and replace servers freely. This is required for horizontal scaling and auto-scaling.
Q3What is wrong with sticky sessions?
They tie users to servers. So the load becomes uneven, users lose their sessions when their server dies, and scaling in or deploying means waiting for attached users to finish. Use them only when needed, such as for some long-lived connections.
Q4You added 10 servers but throughput only doubled. Why?
A shared bottleneck now limits everyone. It is usually the database, a cache, a lock or a third-party API. Amdahl's law says that the part of the work that cannot be done in parallel limits the total speed-up. Find the bottleneck with metrics and remove it (with caching, replicas, batching or partitioning).
Q5Why pre-scale before a big known event instead of relying on auto-scaling?
Auto-scaling reacts after the load rises, and new instances take minutes to start and warm up. When millions of users arrive within minutes, such as for a match or a sale, waiting for the CPU load to climb is too late. So you add capacity in advance.
Key Takeaways
- Vertical scaling means a bigger machine: simple, but limited, expensive at the top end, and a single point of failure.
- Horizontal scaling means more machines behind a load balancer: nearly unlimited and fault-tolerant, but it needs stateless design.
- State (sessions, local files, in-memory data) is what stops apps from scaling out. Move it to Redis, object storage or the database.
- Avoid sticky sessions unless you truly need them.
- Databases usually scale vertically first. Scaling well means finding and removing shared bottlenecks.
Further Reading
- The Twelve-Factor App (factor VI, "Processes")
- The System Design Primer on GitHub (sections "Horizontal scaling" and "Application layer")
- System Design Interview: An Insider's Guide by Alex Xu (chapter 1)
- The Harvard CS75 scalability lecture by David Malan
- Let's Encrypt's blog post about its next-generation database servers
Published by Jainik · Last updated · How these lessons are made
Keep learning
Part of the study guide
Related series