Latency, Throughput & Why p99 Matters

Latency

Latency is the time from sending a request to getting the response. For a web request, it has three parts:

• Network time: the travel time between client and server. Distance matters (see lesson 1).
• Queue time: waiting in line because the server is busy with other requests.
• Service time: the real work: running code, asking the database, calling other services.

You will often hear response time used to mean the same thing.

Throughput

Throughput is how much work the system finishes in a unit of time. It is often measured as:

• RPS/QPS: requests or queries per second,
• TPS: transactions per second (a transaction is one complete business action, such as a payment),
• MB/s: data per second.

Bandwidth is the capacity (the size of the pipe). Throughput is what you really get through it. Difference in one line: bandwidth is what is possible, throughput is what happens.

Little's Law: the link between them

Little's Law is a simple formula that links the number of requests inside a system to the arrival rate and the time each request spends there. L is the average number of requests in the system. λ (lambda) is the arrival rate. W is the average time one request spends in the system.

Requests in the system at once  =  Arrival rate  ×  Time each request spends            L                   =       λ        ×          W

Example: your API gets 500 requests per second, and each takes 40 ms (0.04 s).

L = 500 × 0.04 = 20 requests in flight at any moment

So you need at least about 20 worker threads or connections to keep up. (A worker thread is one unit of work that handles one request.) If the database slows down and requests now take 400 ms, you suddenly have 200 in flight. A pool of 50 threads cannot cope. Requests wait in a queue, and latency climbs even more.

This is how a slow database turns into a full outage.

Why latency explodes when servers get busy

When a server is lightly used, requests are handled right away. As it gets busier, requests start waiting in a queue. Utilisation means how busy the server is, as a percentage. The link between utilisation and latency is not a straight line:

Latency
  │                                    ╱
  │                                  ╱
  │                               ╱      ← near 100%: the queue grows without limit
  │                          __╱
  │_______________________╱              ← headroom: run here (50–70%)
  └──────────────────────────────────── Utilisation
   0%        50%        70%     90% 100%

At 50% busy, waits are short. At 90%, they are much longer. Near 100%, the queue grows without limit. This is why teams usually run servers at around 50–70% CPU, not 95%. The spare capacity (headroom) absorbs bursts, which are short sudden rises in traffic.

Think of a supermarket checkout. When the cashier is idle half the time, you walk right up. When the cashier is busy 95% of the time, one slow customer creates a long line.

Trading latency for throughput (and back)

• Batching (grouping work, such as sending 100 writes at once) raises throughput, but each item waits for the batch, so latency goes up.
• Caching (keeping a saved copy of an answer) lowers latency by skipping slow work.
• Parallel calls (calling three services at once instead of one after another) lower latency.
• More servers raise throughput, but do not make a single request faster.

Why averages lie

Latency is rarely spread out evenly. Most requests are fast, and a small group is very slow. This group is called the "long tail". The widget below lets you try it:

One 5-second request can hide among 99 fast ones and barely move the average. Also, an average of 80 ms might mean nobody really saw 80 ms: half saw 20 ms and half saw 140 ms.

Percentiles

A percentile is a number that answers: "what latency were X% of requests faster than?"

• p50 (median): half the requests were faster than this. This is the "typical" user.
• p90: 90% were faster, so 1 in 10 was slower.
• p95: 1 in 20 was slower.
• p99: 1 in 100 was slower.
• p99.9: 1 in 1,000 was slower.

Example: 100 requests sorted from fastest to slowest.

• The 50th is p50 = 45 ms.
• The 99th is p99 = 1,200 ms.

"p99 = 1.2 s" means 1% of your requests take over 1.2 seconds. At 1 million requests a day, that is 10,000 slow experiences every day.

Why the tail matters more at scale

Users make many requests. Loading one page might trigger 50 API calls. If each call has a 1% chance of being slow, the chance that the page has at least one slow call is:

1 − 0.99^50  ≈ 40%

So your "rare" p99 becomes something that 40% of page loads feel.

Fan-out makes it worse. Fan-out means one request is split into many requests to other servers. A search request might ask 100 servers in parallel and wait for all of them. Even if each server is slow only 1% of the time, the chance that at least one is slow is:

This is the key idea of Google's well-known paper "The Tail at Scale": in big systems, the rare slow case becomes the common case.

Your heaviest users feel it most. The customer with 5,000 items in their order history, or the seller with the biggest shop, often triggers the slowest queries. These are often your most valuable users.

Common causes of tail latency

• Garbage collection pauses. Garbage collection is when a language runtime (in Java, Go or Node.js, for example) stops to free memory that is no longer used.
• Queueing (waiting in line) during short bursts of traffic.
• Cache misses: the saved copy was not there, so the data had to come from the database.
• "Noisy neighbours": other programs on the same machine that use up the shared CPU, disk or network.
• Retries: sending a request again after a timeout (a timeout is when you stop waiting).
• Big users with big data, as described above.
• Cold starts: a new server or serverless function (code that the cloud runs on demand) that has to start up before it can answer.

Techniques to cut the tail

• Timeouts, so that one slow call cannot hang forever (Part 7).
• Caching hot data.
• Hedged requests: if a response has not arrived by the p95 time, send a second copy of the request to another server and use whichever answer comes first. Google describes this idea in its tail-latency paper.
• Keep servers below saturation (saturation means fully busy; keep the headroom from above).
• Paginate or limit heavy queries (return results in small pages), so that no single request does unlimited work.

Measuring percentiles correctly

• Record latencies in histograms. A histogram counts how many requests fell into each time range (for example 0–10 ms, 10–20 ms). Tools like Prometheus, Datadog and HdrHistogram support this.
• You cannot average percentiles. If server A has p99 = 100 ms and server B has p99 = 900 ms, the combined p99 is not 500 ms. Combine the raw histograms, then compute the percentile.
• Measure from the user's side too, not only the server's side. The network and the browser add time.