Metrics and Prometheus: Counters, Gauges, Histograms and Percentiles

Logs told us about one request. But when the phone rings at 2 a.m., the first question is not about one request. It is "how bad is it, right now, for everybody?" For that you need numbers, kept over time, cheap enough to keep forever. That is what metrics are.

This is lesson 2 of nine. In lesson 1 we wrote good logs. Here we build the numbers side with prom-client, a metrics library for Node.js, and Prometheus, a very popular open-source system that collects and stores metrics. I downloaded the real Prometheus binary and ran it against a real Node service, so everything below is pasted from an actual run. The next lesson, Distributed Tracing With OpenTelemetry, adds the third pillar.

You should know basic Node.js and what an HTTP request is. A little math helps for percentiles but I explain it as we go. If the event loop is new, see The Event Loop, because we use event loop lag as a metric.

Words used in this lesson

• A metric is a named number measured over time.
• A time series is one metric with one exact set of labels, plus all its values over time.
• A sample is one value of a time series at one moment.
• A label is a key and value that splits a metric into parts, like status="500".
• Cardinality is how many different time series a metric makes. It equals the number of label combinations in use.
• A counter is a number that only goes up (or resets to zero when the program restarts).
• A gauge is a number that can go up and down.
• A histogram counts observations into buckets, like "how many requests took under 0.1 seconds".
• A bucket is one range of the histogram. In Prometheus a bucket counts everything less than or equal to its upper limit, called le.
• A percentile (or quantile) is a value that a share of observations fall under. The 99th percentile (p99) is the value that 99 percent of requests are faster than.
• Scraping is when Prometheus fetches the /metrics page of your service over HTTP, on a schedule.
• Prometheus is a monitoring system that scrapes metrics and stores them in a time series database.
• PromQL is the Prometheus query language.
• A target is one thing Prometheus scrapes. The built-in metric up is 1 when the last scrape worked, 0 when it did not.
• An exporter is a small program that exposes metrics for something that does not do it itself.
• A recording rule is a saved PromQL query whose result Prometheus stores as a new metric.

What a metric looks like

A metric is just a line of text. Prometheus-style services expose a page, usually /metrics, in a simple text format. Here is a piece of the real page from my checkout service after about 3,000 requests:

# HELP http_requests_total Finished HTTP requests# TYPE http_requests_total counterhttp_requests_total{method="GET",route="/checkout",status="200"} 2941http_requests_total{method="GET",route="/checkout",status="500"} 59# HELP http_requests_in_flight Requests being handled right now# TYPE http_requests_in_flight gaugehttp_requests_in_flight 0

Each line has: a name, optional labels in curly braces, and a value. The lines starting with # HELP and # TYPE describe the metric. Notice that one metric name made two separate lines because the label status has two values: 200 and 500. Each line is its own time series.

Prometheus visits this page every few seconds, reads the numbers, adds the time, and stores them. This is the pull model: Prometheus pulls from your service, rather than your service pushing. Pulling has real benefits. Prometheus knows when a target is down (the scrape fails and up becomes 0). Your service does not need to know where the monitoring system is. And you can open /metrics in a browser to see what is exposed. (For short batch jobs that finish before a scrape can happen, the Prometheus docs describe pushing to a Pushgateway. That is the exception.)

The first metric type: counter

A counter counts events. It starts at zero and only goes up: requests handled, errors, bytes sent, orders placed. It never goes down, except that it resets to zero when the process restarts.

import client from "prom-client";const register = new client.Registry();const reqs = new client.Counter({  name: "http_requests_total",  help: "Finished HTTP requests",  labelNames: ["method", "route", "status"],  registers: [register],});reqs.inc({ method: "GET", route: "/checkout", status: 200 });

The raw counter value is almost never useful by itself. "2941 requests so far" does not tell you if now is busy or quiet. What you want is the rate: how fast it is growing. PromQL has a function for this called rate(). I asked my real Prometheus:

sum(rate(http_requests_total[20s]))     ->  210.5   (requests per second)

rate(x[20s]) means "the average per-second increase of x over the last 20 seconds". The [20s] part is the window. For a normal 15-second scrape interval you use a window of at least 1 to 2 minutes, so that the window holds several scrapes. The Prometheus docs also recommend that the metric name for a counter ends with _total.

Counters survive restarts (when you use rate)

What happens to a counter when your service restarts? It goes back to zero. A naive chart would show a big drop. Prometheus functions like rate() and increase() know about this and treat a drop as a reset. I tested it by restarting my service in the middle of a run. The total of all request counts over time, queried at 20-second steps, looked like this:

7000 7000 7000 7000 7000 7000 300

The raw counter fell from 7000 to 300 after the restart. Then I asked:

resets(http_requests_total{status="200"}[10m])   ->  1sum(increase(http_requests_total[10m]))          ->  7268.4

resets() counted one reset, and increase() still gave about 7,268 for the 10-minute window, close to the true 7,300 (7,000 before plus 300 after). It is not exactly 7,300 because Prometheus extrapolates at the edges of the window. Do not expect increase() to be an exact integer. It is a good estimate.

The second type: gauge

A gauge is a number that can go up or down: temperature, memory in use, queue length, requests being handled right now. It is a snapshot of "how much now".

const inflight = new client.Gauge({  name: "http_requests_in_flight",  help: "Requests being handled right now",  registers: [register],});inflight.inc();   // request startsinflight.dec();   // request ends

Because a gauge is only a snapshot, you can miss peaks between scrapes. If the queue length jumps to 5,000 for 3 seconds and you scrape every 15 seconds, you may never see it. For things that matter, either scrape more often, or also keep a counter or histogram that captures the events.

How to pick between counter and gauge? Ask: "does it only ever grow, and do I care about how fast?" Counter. "Does it go up and down, and do I care about the current level?" Gauge. A common mistake is to use a gauge for something like "total errors", and then lose the history on restart, or lose events between scrapes.

Free metrics: the default collectors

Call client.collectDefaultMetrics({ register }) and prom-client adds process metrics with no work from you: CPU time (process_cpu_seconds_total), memory, open file handles, garbage collection timings, and Node-specific numbers such as event loop lag. In my run a scrape showed:

process_cpu_seconds_total 2.974952nodejs_eventloop_lag_seconds 0.001764259nodejs_heap_size_used_bytes 9618112

That is about 1.8 ms of event loop lag, and 9.6 MB of heap in use. These are your saturation signals from lesson 0. If the event loop lag goes up to hundreds of milliseconds, your single JavaScript thread is too busy.

The third type: histogram

Now the interesting one. We want to know how long requests take. A single number is not enough.

Why averages lie

I made the checkout service slow for 5 percent of requests (600 to 900 ms) and fast for the rest (20 to 80 ms). Then I fired 4,000 requests at it from a load script and recorded each latency on the client. Sorted, the numbers say:

average: 0.0815 s   (81 ms)p50:     0.0533 s   (53 ms)p95:     0.0884 s   (88 ms)p99:     0.8199 s   (820 ms)max:     0.8999 s   (900 ms)

Look at the average: 81 ms. It looks healthy. But 1 request in 100 took 820 ms or more, which is ten times longer. An average mixes the fast majority and the slow tail into one calm number. Nobody experiences "the average request". A user who waits 820 ms is not comforted by the fact that the average is 81 ms.

Percentiles fix this. p50 is the median: half of requests are faster. p95 and p99 describe the tail. The usual approach is to watch p50 for the typical case and p95 or p99 for the bad experiences. Notice also that p95 here (88 ms) still looks fine, because only 5 percent of requests are slow. If your slow share were 6 percent, p95 would jump. Always know which percentile hides which problem.

But how can a metric keep percentiles?

You cannot keep all 4,000 latencies in a metric. That would be a log. A histogram solves this with a trick: you choose buckets (ranges) ahead of time, and keep one counter per bucket. Each request increments the counters of the buckets it fits in. In prom-client:

const dur = new client.Histogram({  name: "http_request_duration_seconds",  help: "Request duration",  labelNames: ["route"],  buckets: [0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5],  registers: [register],});const end = dur.startTimer({ route: "/checkout" });// ... handle the request ...end();    // records the elapsed seconds

And this is what shows up on /metrics after 3,000 requests (real output):

http_request_duration_seconds_bucket{le="0.01",route="/checkout"} 0http_request_duration_seconds_bucket{le="0.025",route="/checkout"} 239http_request_duration_seconds_bucket{le="0.05",route="/checkout"} 1435http_request_duration_seconds_bucket{le="0.1",route="/checkout"} 2849http_request_duration_seconds_bucket{le="0.25",route="/checkout"} 2849http_request_duration_seconds_bucket{le="0.5",route="/checkout"} 2849http_request_duration_seconds_bucket{le="1",route="/checkout"} 3000http_request_duration_seconds_bucket{le="2.5",route="/checkout"} 3000http_request_duration_seconds_bucket{le="+Inf",route="/checkout"} 3000http_request_duration_seconds_sum{route="/checkout"} 256.6344777460004http_request_duration_seconds_count{route="/checkout"} 3000

How to read it: the buckets are cumulative. The bucket le="0.1" (less than or equal to 0.1 s) holds 2,849, which includes everything faster too. le="+Inf" holds all 3,000. Two more lines complete the histogram: _count (how many observations, 3,000) and _sum (the total of all durations, 256.6 s). Dividing sum by count gives the average, 256.6 / 3000 = 0.0855 s.

You can see the shape of the problem right in the numbers: 2,849 requests finished within 100 ms, and then the counts do not change at 0.25 and 0.5, which means no request took between 100 ms and 500 ms. The last 151 requests (3000 minus 2849) took between 0.5 and 1 s. That is the 5 percent slow group.

Getting a percentile from buckets

Prometheus uses the function histogram_quantile(). I asked for p50, p95 and p99 over the last minute, from my real server:

histogram_quantile(0.5,  sum by (le) (rate(http_request_duration_seconds_bucket[1m])))   ->  0.0525histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[1m])))   ->  0.0997histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[1m])))   ->  0.8945

Compare with the exact values from the client: p50 0.0533, p95 0.0884, p99 0.8199. The histogram answers are close, but not equal. p99 is 0.8945 against a true 0.82. Why?

Because Prometheus only knows how many requests fell in each bucket, not where inside. For p99 it finds the bucket where the 99th percentile falls (here between 0.5 and 1 s), and assumes the values are spread evenly inside it, then interpolates a straight line. Let me do the math by hand with the earlier bucket numbers (3,000 requests): the p99 rank is 0.99 × 3000 = 2970. The bucket below has 2,849 (at 0.5 s) and the bucket above has 3,000 (at 1 s). So the estimate is 0.5 + (1 − 0.5) × (2970 − 2849) / (3000 − 2849) = 0.5 + 0.5 × 121 / 151 = 0.9007. That is the same sort of answer as Prometheus gave (0.8945 on a different window). The real values in that bucket are between 0.6 and 0.9, so the guess is a little off.

The estimate is always only as good as your buckets. To show this, I generated 100,000 fake latencies with the same shape and estimated percentiles with three bucket layouts, using the same interpolation:

p50  exact=0.0514    default-ish [.005..10]                                       0.0524    coarse [0.1,1,10]                                            0.0526    tight around SLO [..0.05,0.1,0.25,0.5,0.6,0.7,0.8,0.9,1]     0.0524p95  exact=0.6003    default-ish [.005..10]                                       0.5006    coarse [0.1,1,10]                                            0.1011    tight around SLO [..0.05,0.1,0.25,0.5,0.6,0.7,0.8,0.9,1]     0.6004p99  exact=0.8383    default-ish [.005..10]                                       0.9001    coarse [0.1,1,10]                                            0.8202    tight around SLO [..0.05,0.1,0.25,0.5,0.6,0.7,0.8,0.9,1]     0.8390average: 0.0850

(This was my own simulation of the estimate formula, not Prometheus itself. It reproduces what the real server did on the hand check above.) Read the p95 row: the true value is 0.60 s, but with the coarse buckets the estimate was 0.10 s, six times too low. The coarse layout hid the entire slow tail from p95. That is a dangerous failure: the chart is smooth and wrong. The p50 is fine in all three layouts because the typical value is in a well-covered area.

How to choose buckets:

• Put more buckets near the numbers that matter, especially around your target (the SLO threshold in lesson 4). If your goal is "99 percent under 300 ms", you need a bucket edge at or very near 0.3 s.
• Use roughly exponential spacing (1, 2.5, 5, 10, 25 ...), as the defaults do, to cover orders of magnitude.
• Do not add dozens of buckets. Each bucket is a separate time series for each label combination.
• Do not put a bucket for every millisecond. Think of the buckets as your resolution.

Summary, and the newer native histograms

Prometheus has a fourth type, the summary. A summary computes percentiles inside your program and exposes them directly. They are accurate for that one process, but you cannot add them up across servers, and you cannot choose a different percentile later. For most services that run on more than one instance, the histogram is the better choice.

The Prometheus docs go further and say that if you can, prefer native histograms over both classic histograms and summaries. A native histogram is a newer representation, where one sample holds the whole distribution with automatically sized buckets, so you do not choose buckets by hand. The docs also say that support in client libraries is still limited and that exposing them currently needs the protobuf format. So check what your library supports. Everything in this lesson about interpreting percentiles remains true.

Labels and cardinality: the bill arrives here

Labels are powerful. With status and route labels I can ask for errors on the checkout route only. But each distinct combination of label values is a separate time series, and the cost grows by multiplication.

Say you have 3 methods, 50 routes and 10 status codes. Worst case that is 3 × 50 × 10 = 1,500 series for one counter. For a histogram with 10 buckets add a factor of about 12 (each bucket plus sum and count). That is 18,000. Still OK. Now someone adds a user_id label with 100,000 users: the number explodes.

I tested this directly. Two counters, both updated 10,000 times:

const good = new client.Counter({ name: "orders_total", help: "h", labelNames: ["status"] });const bad  = new client.Counter({ name: "orders_by_user_total", help: "h", labelNames: ["status", "user_id"] });for (let i = 0; i < 10000; i++) {  good.inc({ status: i % 20 ? "ok" : "failed" });  bad.inc({ status: i % 20 ? "ok" : "failed", user_id: "u" + i });}

orders_total series: 2orders_by_user_total series: 10000/metrics size bytes: 521072

Two series versus ten thousand, from the same events. The page that Prometheus has to scrape grew to 521 KB. Prometheus has to keep every series in memory for the recent data. A label like this is the most common way to make a metrics system slow, expensive or to crash it.

The Prometheus instrumentation guide gives numbers as a rule of thumb. It suggests keeping the cardinality of your metrics below 10 (and says the vast majority of metrics should have no labels at all), and says that if a metric has a cardinality over 100 or the potential to grow that large, you should look for another approach, for example a log or trace for the high-detail part. Treat it as a guide, not a law. Some well-known metrics have hundreds of series and work fine.

Safe label values are: HTTP method, a route template (/users/:id, not /users/123), status code or class, region, a service version. Dangerous: user id, email, request id, full URL with query string, raw error messages, timestamps, IP addresses.

Another rule: labels should be low-cardinality dimensions that you really want to slice by. Details about one request belong in logs and traces. That is the split that makes the whole system affordable. A metric tells you that p99 for /checkout in region eu got worse. The trace and log for one slow request tell you why.

Naming rules that help

The Prometheus naming guide is short and worth reading. In my own words, the main points:

• Use a short prefix for the domain, like http_ or shop_.
• Use base units: seconds not milliseconds, bytes not kilobytes. Put the unit in the name as a suffix: _seconds, _bytes.
• A counter ends with _total: http_requests_total. A counter with a unit: process_cpu_seconds_total.
• A name must describe a single thing. A good test from the guide: the sum or the average over all label values should make sense. Do not mix "queue capacity" and "queue length" in one metric with a label.
• Do not put label names in the metric name, or the other way round.

Following these makes your dashboards and your PromQL easier to share, and it avoids unit mix-ups (the famous bug where one service reports milliseconds and another seconds).

Running a real Prometheus

Here is the minimal setup I used. A config file prom.yml:

global: { scrape_interval: 1s, evaluation_interval: 1s }scrape_configs:  - job_name: shop    static_configs: [{ targets: ["localhost:3100"] }]

scrape_interval: 1s is just for my short demo; use 15s or 30s in real life, since one scrape that fast is wasteful. I checked the file, then started the server:

$ promtool check config prom.ymlChecking prom.yml SUCCESS: prom.yml is valid prometheus config file syntax$ prometheus --config.file=prom.yml --storage.tsdb.path=./data --web.listen-address=:9190

(I used port 9190 because the default 9090 was in use on my machine.) Then I checked that it was scraping:

up{instance="localhost:3100", job="shop"} = 1target health: up  http://localhost:3100/metrics  interval 1s

up = 1 means the last scrape succeeded. Alerting on up == 0 is the simplest "this service is down" alert there is.

Then I sent load and queried through the API. You can use the web page at http://localhost:9190, curl, or promtool query instant:

$ promtool query instant http://localhost:9190 'sum(rate(http_requests_total[30s])) by (status)'{status="200"} => 135.82758620689654 @[1791776662.796]{status="500"} => 2.103448275862069 @[1791776662.796]

That is 135.8 successful requests per second and 2.1 failed per second. Dividing: 2.1 / (135.8 + 2.1) is about 1.5 percent errors. The service was configured to fail 2 percent of the time at random, and over the whole run the measured value was 1.7 percent (120 failures out of 7,000). Small samples wobble.

A few PromQL ideas that cover 90 percent of use

# traffic (requests per second)sum(rate(http_requests_total[5m]))# error ratiosum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))# p99 latency across all instanceshistogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))# average latency (from the sum and count lines)rate(http_request_duration_seconds_sum[5m]) / rate(http_request_duration_seconds_count[5m])# by routesum by (route) (rate(http_requests_total[5m]))

Four things to notice. rate() always takes a range in square brackets. sum by (...) groups and adds the series. status=~"5.." is a regular expression match for any 5xx status. And histogram_quantile needs the le label to stay in the grouping.

A practical warning about rate() windows (my own rule of thumb, not from the docs I read): the window should hold at least a few scrapes. With a 15-second scrape interval, a [1m] window gives about four samples, and [5m] is a common choice for stable dashboards.

Recording rules

Some queries are heavy or used in many places (dashboards and alerts). A recording rule saves the result of a query as a new metric. The ones I used for the SLO work in lesson 4 look like this:

groups:- name: checkout-slo  rules:  - record: sli:errors:ratio_rate1m    expr: sum(rate(http_requests_total{status=~"5.."}[1m])) / sum(rate(http_requests_total[1m]))

promtool check rules accepts the file (SUCCESS: 4 rules found for my four rules). The sli:errors:ratio_rate1m name follows the community habit of level:metric:operations, with colons marking a recorded rule. When the service ran under load, this recorded series showed an error ratio of about 0.0162 (1.6 percent).

Counters, gauges, histograms: which to use?

What goes wrong in practice

• Too many labels (cardinality blow-up). Covered above. Check the size of /metrics after adding a label.
• Wrong buckets. The p95 example showed a 6x error from coarse buckets. After you choose an SLO threshold, check that a bucket edge sits at it.
• Counting at the wrong moment. The Prometheus docs say to be consistent about counting a request when it starts or when it ends, and suggest "when it ends", because that lines up with errors and latency.
• Exposing /metrics to the public internet. Metrics can show internal names and sizes. Serve it on an internal port or behind network rules.
• Using averages for latency. We have seen enough of that.
• Gauges that hide spikes. Use a counter or a histogram for events that matter.
• No metric on failures. Count successes and failures, not only successes. With a status label on one counter you can do both.
• A metric that disappears. If a label combination has never occurred, no line exists. A rate() of a missing series gives nothing instead of 0. For important counters, initialise them (counter.inc(labels, 0)) so that the series exists from the start.

Try it

1. Install prom-client. Make a small HTTP server with /metrics. Add a counter for requests with labels route and status, and a histogram for duration. Fire 1,000 requests with a loop and look at the page.
2. Make 5 percent of requests take 500 ms and the rest 20 ms. Using only the numbers on the /metrics page, find out how many requests took 100 ms or less, and estimate p99 by hand with the interpolation formula.
3. Add a label user_id to a counter and send 5,000 requests with different ids. How many lines does the page have now? Remove the label again.
4. Why can you not graph http_requests_total directly and read "requests per second"? What do you use instead?
5. You have 3 servers. Each reports p99 latency as a gauge of 120 ms, 130 ms and 800 ms. Can you say the overall p99 is 350 ms (the average)? What should you have exposed instead?
6. Which of these are safe labels? route="/users/:id", email="a@b.com", status="503", path="/users/8231?x=1", region="eu".

Answers

1. You should see lines such as http_requests_total{route="/",status="200"} 1000 and a set of _bucket, _sum and _count lines for the histogram.
2. The line for the bucket le="0.1" is the count of requests at 100 ms or less (about 950 of 1,000). With the slow ones at 500 ms and a bucket edge near 0.5, the estimate for p99 will land close to the upper range. The formula is lower + (upper − lower) × (rank − lowerCount) / (upperCount − lowerCount), where rank is 0.99 × total. The estimate depends on your buckets, and so does its error.
3. About 5,000 more lines, one per user id, plus the page gets much bigger. Remove the label to bring it back to a few lines.
4. The raw counter is a total since the process started, and it resets on restart. You use rate(http_requests_total[5m]), which gives the per-second increase and handles restarts.
5. No. You cannot average percentiles, and the overall p99 depends on how many requests each server handled. Expose a histogram on each server, sum the buckets with sum by (le) (rate(...)) and compute histogram_quantile on the sum.
6. Safe: route="/users/:id", status="503", region="eu". Unsafe: email (unbounded and personal data), and the raw path with an id and query string (unbounded).

Interview questions

Key takeaways

• A counter only goes up, a gauge goes up and down, and a histogram counts values into buckets. Query counters with rate(), never raw.
• Averages lie. In my real run the average latency was 81 ms and p99 was about 820 ms.
• Percentiles from histograms are estimates. They are only as good as the buckets. Coarse buckets showed p95 as 0.10 s when the truth was 0.60 s.
• Never average percentiles. Sum histogram buckets across servers, then compute the quantile.
• Labels multiply series. One user_id label turned 2 series into 10,000. Use route templates and other bounded labels.
• Counters reset on restart. rate() and increase() handle this. Raw values do not.
• Prometheus pulls /metrics on a schedule and gives you up for free. Check your configs with promtool.
• Name metrics with base units and _total, and initialise important counters so the series exist.

What's next

Next: Distributed Tracing With OpenTelemetry, Lesson 3. Metrics tell you that p99 is high. A trace tells you which step in which service is the slow one. You will instrument two real local services, watch the trace context travel in an HTTP header, and read a trace to find a slow database call. Practise now: add the three metric types to a service of your own, and check how many lines your /metrics page has.