System Design, Step by Step

Part 3 · The Building Blocks

Load Balancers: L4 vs L7 and Balancing Algorithms

One address, many servers, and one of them just crashed at 2 AM. How load balancers spread traffic, L4 vs L7, the balancing algorithms and when each wins, and why a “deep” health check can take everything down at once.

Lesson 1210 min readbeginner#load-balancer#scalability

On this page

  1. The Problem
  2. The Core Idea
  3. How It Works
  4. Trade-offs
  5. In the Real World
  6. Interview Questions
  7. Key Takeaways
  8. Further Reading

The Problem

You've made your app stateless (post 7) and started three servers. But users can only type one address into their browser. How does traffic get spread across three machines? And what happens when server 2 crashes at 2 AM? Without help, a third of your users would see errors until someone wakes up and fixes it.

The load balancer solves both problems. It's the piece that makes horizontal scaling and high availability actually work.

A load balancer is a server (or a service) that receives all incoming requests and shares them between several servers behind it. Horizontal scaling means adding more servers instead of buying one bigger server. High availability means the system keeps working even when some parts fail.

The Core Idea

Picture the token system at a busy bank or passport office. You don't choose a counter yourself. You take a token, and the display tells you which counter is free. If counter 4 closes for lunch, the system simply stops calling people to counter 4.

A load balancer does the same for requests:

  • It sits in front of your servers, with one public address.
  • It sends each request to a healthy server.
  • It spreads the work so no single server is overloaded.
  • It stops sending traffic to servers that fail health checks. A health check is a small test request that shows if a server is alive.
Diagram
                   ┌──► [ Server 1 ] ✅
 Users ──► [ LB ] ─┼──► [ Server 2 ] ❌  (failed health check, no traffic)
                   └──► [ Server 3 ] ✅

How It Works

What a load balancer does

  1. Distributes traffic across servers using an algorithm.
  2. Health-checks servers and routes around failures.
  3. Hides your internal setup. Users see one address, not 50 servers.
  4. Enables zero-downtime deploys. A deploy is the act of releasing a new version. Take servers out one at a time, update them, put them back, and users never see a break.
  5. Often terminates TLS (HTTPS), so app servers don't have to (post 3). TLS is the encryption layer, and "terminate" means the load balancer decrypts the data and sends plain requests on to the servers.

Layer 4 vs Layer 7

Remember the network layers from post 1: Layer 4 is transport (TCP/UDP) and Layer 7 is application (HTTP). TCP and UDP are two ways to send data across a network. TCP is reliable and UDP is faster but does not promise delivery. A port is a number that tells the computer which program should get the data.

L4 load balancer. It looks only at IP addresses and ports. It doesn't read the request; it forwards the connection's packets to a chosen server.

  • ✅ Very fast, and can handle huge numbers of connections.
  • ✅ Works with any protocol on TCP/UDP: databases, game servers, MQTT, gRPC.
  • ❌ Can't make decisions based on the URL, headers or cookies.

L7 load balancer. It reads the HTTP request: path, host, headers, cookies.

  • ✅ Smart routing: /api/* → API servers, /images/* → image servers, admin.site.com → admin servers.
  • ✅ Can retry failed requests, add headers, compress responses, and route by user or region.
  • ✅ Balances per request, not per connection. This matters for HTTP/2 and gRPC, where one connection carries many requests.
  • ❌ More work per request, so slightly slower and more expensive than L4.
  • ❌ Must decrypt HTTPS to read requests, so it needs your TLS certificate.
L4 — transport
  • •Sees: “packet for 203.0.113.5:443”
  • +Very fast; huge connection counts
  • +Any TCP/UDP protocol: databases, games, MQTT
  • −Can't route by URL, header or cookie
  • −Balances connections, not requests
Use whenAWS NLB; the first layer at the edge
L7 — application
  • •Sees: “GET /api/orders, Host: shop.com, Cookie: …”
  • +Route /api, /images and admin.site.com to different pools
  • +Per-request balancing (matters for HTTP/2, gRPC)
  • +Retries, header rewrites, compression
  • −More work per request; must hold your TLS cert
Use whenAWS ALB, NGINX, Envoy
What each kind of load balancer can see — and therefore decide on. · compare
L4L7
SeesIP, portFull HTTP request
Routing by URL/header❌✅
SpeedFasterSlightly slower
ProtocolsAny TCP/UDPHTTP, HTTP/2, gRPC, WebSocket
AWS exampleNetwork Load Balancer (NLB)Application Load Balancer (ALB)

Many large systems use both: an L4 layer at the edge to spread raw traffic across a fleet of L7 proxies, which then do smart routing.

Balancing algorithms

An algorithm here is a rule for choosing which server gets the next request. Round robin. Send requests in turn: 1, 2, 3, 1, 2, 3…

  • Simple and fair when all servers and requests are similar.
  • Bad when some requests are heavy (a report) and others light (a ping).

Weighted round robin. Bigger servers get a bigger share. A server with weight 3 gets three requests for every one sent to a server with weight 1. Useful when machines differ, or to send a small share to a new version (a canary release: a few users try the new version first).

Least connections. Send to the server with the fewest active connections. It adapts to uneven request times, which makes it good for long or variable requests. One caveat: a newly added server has zero connections, so it can get flooded all at once while its caches are still cold. Many load balancers offer slow start, which ramps up traffic to new servers gradually.

Least response time. Pick the server that has been answering fastest recently. Good for latency, but more complex.

IP hash / consistent hashing. A hash function turns any input (like an IP address) into a number. Use that number to pick a server, so the same key always goes to the same server.

  • Useful when a server keeps a local cache per user or per item, which improves cache hit rates.
  • Consistent hashing (Part 4) is a way of hashing where adding or removing a server only moves a small share of keys.

Power of two random choices. Pick two servers at random, then send the request to the less busy one. It sounds too simple, but it works remarkably well at large scale. It avoids everyone rushing to the same "least busy" server at once, a problem called herding, and it needs little coordination. It's used in several modern proxies (a proxy is a server that passes requests on for others).

Quick guide:

  • Similar requests, similar servers → round robin.
  • Variable request times → least connections, or power of two choices.
  • Need the same key on the same server → consistent hashing.
  • Mixed server sizes → weighted.
Load
balancer
S1active 0 · total 0
S2active 0 · total 0
S3active 0 · total 0

Takes turns: fair by count, blind to how busy each server really is. Watch a server pile up slow requests.

Send requests through four algorithms. About a third of them are slow and stay active for a while — watch where they pile up. · interactive

Health checks

The deep health check trap

If every server's /health also checks the shared database, one short database blip makes every server fail at once, and the load balancer removes all of them. A tiny hiccup becomes a total outage. Check the server itself, handle dependency failures with timeouts and fallbacks, and configure the balancer to fail open when everything looks unhealthy.

The load balancer regularly asks each server, "Are you OK?"

  • Active checks. The LB calls something like GET /health every few seconds. After, say, 3 failures in a row, the server is marked unhealthy. After a couple of successes, it's added back.
  • Passive checks. The LB watches real traffic. If a server keeps returning errors or timing out, it's removed.

Shallow vs deep health checks:

  • A shallow check ("the process is running and can respond") is fast and safe.
  • A deep check ("I can reach the database, the cache and the payment API") sounds better, but it's dangerous. If the shared database has a short hiccup, every server fails its check at once, and the load balancer removes all of them. A small blip becomes a total outage.

A common approach: health checks confirm the server itself is OK. Dependency problems are handled separately, with timeouts and fallbacks (Part 7). Many load balancers also have a "fail open" safety rule: if every server looks unhealthy, keep sending traffic anyway.

Kubernetes is a system that runs and manages many app containers for you. There you'll see the same ideas as liveness probes ("should I restart this?") and readiness probes ("should I send it traffic?").

Connection draining

  1. t = 0 s
    server marked “draining” — no new requests are sent to it
  2. 0–30 s
    in-flight requests finish normally
  3. t = 30 s
    removed from the pool; safe to stop and update
  4. after deploy
    passes health checks → added back, with slow start
Removing a server for a deploy without cutting anyone off. · timeline

When you remove a server for a deploy, you don't want to cut off requests that are halfway through. Connection draining (also called deregistration delay) means: stop sending new requests, let existing ones finish for up to N seconds, then remove the server.

Sticky sessions

Sticky sessions mean the load balancer keeps sending the same user to the same server. As in post 7, the LB can do this with a cookie (a small piece of data the browser sends with each request). This helps with legacy stateful apps, but it causes uneven load and lost sessions when that server dies. Prefer stateless apps.

Isn't the load balancer a single point of failure?

Yes, if there's only one. Common solutions:

  • A pair of load balancers (active-passive) sharing a "floating" IP address that moves to the backup if the main one dies.
  • Multiple load balancers behind DNS or anycast.
  • Managed cloud load balancers (AWS ALB/NLB, Google Cloud Load Balancing), which are already spread across many machines and zones for you.

Global vs local load balancing

  • Global: choose the region or data centre (using GeoDNS, anycast or a global load balancer), as in post 11.
  • Local: inside a data centre, choose the server.

A request usually goes through both.

  • NGINX and HAProxy: open-source, extremely widely used, and can do L4 and L7.
  • Envoy: a modern L7 proxy with advanced balancing, retries and observability (tools to see what the system is doing). It's the base of many service meshes (a layer that manages how services talk to each other).
  • Cloud: AWS ALB (L7) and NLB (L4), Google Cloud Load Balancing, Azure Load Balancer and Application Gateway.
  • Kubernetes: Services and Ingress controllers use these same ideas.

Trade-offs

  • L4 is fast and simple but "blind". L7 is smart but costs more per request, and needs TLS termination.
  • Smarter algorithms (least response time) react better, but need more tracking and can oscillate. Simple ones are predictable.
  • Aggressive health checks find failures fast, but can remove servers that were only briefly slow. Relaxed ones are stable but slower to react.
  • Sticky sessions are convenient for stateful apps, but cause uneven load and weaker failover.
  • Each LB layer adds a network hop and a little latency.

When not to use a load balancer: a small app on a single server. A load balancer in front of one server adds cost without adding much, until you add a second server or need zero-downtime deploys.

In the Real World

Google's Maglev. Google built its own software load balancer, called Maglev, which runs on ordinary servers instead of special hardware. It handles traffic for services like Search and Gmail, and uses a special form of consistent hashing so connections stay on the same backend even as machines come and go. Google published a paper about it in 2016.

GitHub's load balancer. GitHub built and open-sourced its own load balancer system (GLB), designed so that servers can be added or removed, or the load balancers themselves updated, without dropping users' connections.

AWS ALB and NLB. On AWS, most web apps use an Application Load Balancer (L7) to route by path or host to different services. Workloads needing raw TCP/UDP or extreme performance use a Network Load Balancer (L4).

The deep health check trap. AWS has written in its Builders' Library about how health checks that depend on shared resources can cause every server to fail at once. That's why its advice includes "fail open" behaviour and keeping health checks focused on the server itself.

Kubernetes everywhere. If you deploy an app on Kubernetes with three replicas behind a Service, you're using load balancing plus health checks (readiness probes) every day, often without thinking about it.

Interview Questions

Q1What's the difference between an L4 and an L7 load balancer?

L4 forwards connections using only IPs and ports, so it's fast, cheap and protocol-agnostic. L7 terminates the connection and reads HTTP, so it can route by path, host or header, retry, and balance individual requests — at more cost per request, and it needs the TLS certificate.

Q2When would round robin be a bad choice?

When request costs vary widely. A server that happens to get several slow reports keeps receiving its turn anyway and piles up. Least connections, least response time or power-of-two-choices adapt to real load.

Q3Explain “power of two random choices”.

Pick two servers at random and send the request to the less loaded one. It gets close to the ideal least-loaded choice with almost no coordination, and avoids herding — every balancer rushing to the same “least busy” server at once.

Q4Why can deep health checks cause outages?

If they test shared dependencies, a single dependency hiccup fails every server's check at the same moment and the balancer removes the whole fleet. Keep checks shallow, handle dependencies with timeouts and fallbacks, and fail open when all targets look unhealthy.

Q5Isn't the load balancer a single point of failure?

Only if there's one. Run an active-passive pair with a floating IP, several balancers behind DNS or anycast, or a managed cloud balancer that is already spread across machines and availability zones.

Q6What is connection draining?

When a server is being removed, the balancer stops sending it new requests but lets in-flight ones finish for a grace period before taking it out, so deploys and scale-in don't cut users off mid-request.

Key Takeaways

  • A load balancer spreads traffic across servers, removes unhealthy ones, and enables zero-downtime deploys.
  • L4 routes by IP and port (fast, any protocol). L7 reads HTTP (smart routing by URL and headers). Big systems often use both.
  • Choose the algorithm to fit the workload:
    • round robin for simple cases,
    • least connections or power of two choices for uneven requests,
    • consistent hashing to keep the same key on the same server.
  • Keep health checks focused on the server itself. Deep checks on shared dependencies can take everything down at once.
  • Make the load balancer itself redundant (pairs, DNS/anycast, or a managed cloud LB).

Further Reading

  • Site Reliability Engineering by Google (chapters "Load Balancing at the Frontend" and "Load Balancing in the Datacenter")
  • The Maglev paper, "Maglev: A Fast and Reliable Software Network Load Balancer" (NSDI 2016)
  • The System Design Primer on GitHub (section "Load balancer")
  • The AWS Builders' Library article "Implementing health checks"
  • The official HAProxy and NGINX load-balancing documentation

Published by Jainik · Last updated · How these lessons are made

Keep learning

Part of the study guide

Related series