What Observability Is and the Three Pillars

Every team meets this the same way. It is 2 a.m., a phone rings, and the message says "checkout is slow". You open the server. The CPU looks fine. The logs have a million lines. Somebody says "it was fine yesterday". Nobody knows where to look. The code did not get worse in one night. What was missing was a way to ask the running system questions.

This series is about building that way to ask. We will do it with small programs you can run on your own computer. We will not buy any tool. We use Node.js, a log library called pino, a metrics library called prom-client, a real Prometheus server, and the OpenTelemetry SDK. At the end you will also know the reliability side: what a service level objective is, how to alert on it, how to run an incident, and how to build code that fails gently.

This is lesson 0 of nine. It gives you the map: what the words mean, what the three pillars are, what question each one answers, and which few numbers to look at first. The next lesson, Logging That Helps, goes deep on logs. After that come metrics and Prometheus, tracing and SLIs, SLOs and error budgets. Lessons 5 to 8 cover alerting and dashboards, incidents, reliability patterns and interview questions.

You need basic JavaScript and a rough idea of what a web server does. If a request, a status code or a port is new to you, read An HTTP Server From Scratch first. If you want a head start on how a Node service logs and shuts down, Errors, Logging and Graceful Shutdown is a good companion.

Words used in this lesson

• Monitoring is watching a system with checks and dashboards you prepared ahead of time, to tell you when something is wrong.
• Observability is how well you can understand what is going on inside a system by looking at the data it sends out.
• Telemetry is the data a running system emits about itself: logs, metrics and traces.
• A log is a record of one event, with a time, written when something happens.
• A metric is a number measured over time, such as requests per second.
• A trace is the story of one request as it moves through your services.
• A span is one step inside a trace, such as "call the database".
• Instrumentation is the code that makes your program emit telemetry.
• A signal is OpenTelemetry's word for one kind of telemetry (logs, metrics or traces).
• Cardinality is the number of different values a label can take. High cardinality means very many.
• An SLI, an SLO and an error budget are a measurement, a goal for that measurement, and the amount of failure the goal allows. Lesson 4 explains them.
• The four golden signals are latency, traffic, errors and saturation.
• RED is rate, errors and duration, for services. USE is utilisation, saturation and errors, for resources.
• Unknown unknowns are problems you did not predict, so you did not build a check for them.

Monitoring versus observability

People argue about these two words. Let us keep it simple and useful.

Monitoring is when you decide in advance what can go wrong, and you build a check for each thing. "Alert me if CPU is above 90 percent." "Alert me if the error rate is above 2 percent." This works well for problems you have seen before. These are the known unknowns: you know the question, you only need to watch the answer.

Observability is a property of the system, not a tool you install. A system is observable when you can look at the data it already emits and answer a question you did not think of before. "Why are only users in Germany with a specific app version slow?" Nobody wrote a check for that. If your data has enough detail (country, app version, the path of each request), you can slice it and find out. These are the unknown unknowns.

The OpenTelemetry docs describe it in a similar way. Observability lets you understand a system from the outside, by asking questions without knowing its insides, including the "why is this happening" question. They add that an application is properly instrumented when developers do not need to add more code to troubleshoot a problem, because the data they need is already there.

So the two work together. Monitoring is the smoke alarm. Observability is being able to walk into the building and see which room the fire is in.

Three kinds of data

There are three main kinds of telemetry. People call them the three pillars. The name is a bit too neat, because they overlap and none of them is enough alone, but it is a good way to learn.

Let us feel the difference with one story. I wrote a tiny checkout service and added a problem on purpose. About 5 percent of requests are slow (around 600 to 900 ms instead of 20 to 80 ms). About 2 percent fail with status 500. It is the same service we will build properly in lessons 1 to 3. For now, look at the same incident through each pillar.

Pillar 1: logs, "what happened in this case?"

A log is a line of text (or better, a line of JSON) that the program writes when something happens. Here is real output from two of my services handling one request that failed. The gateway called the inventory service, and the inventory service said no:

{"level":30,"service":"gateway","requestId":"from-client-1","method":"GET","url":"/item?sku=bad","msg":"request received"}{"level":30,"service":"inventory","requestId":"from-client-1","method":"GET","url":"/stock?sku=bad","msg":"request received"}{"level":40,"service":"inventory","requestId":"from-client-1","sku":"bad","msg":"sku lookup failed"}{"level":30,"service":"inventory","requestId":"from-client-1","status":500,"durationMs":21,"msg":"request done"}{"level":50,"service":"gateway","requestId":"from-client-1","err":"upstream 500","msg":"handler failed"}{"level":30,"service":"gateway","requestId":"from-client-1","status":500,"durationMs":24,"msg":"request done"}

This is great detail. You can see the exact SKU that failed, which service said what, and in what order. Logs are the best pillar for the question "what exactly happened to this one request?"

But logs are poor at a different question: "how many requests are failing right now?" To answer it you would have to count lines. At 10,000 requests per second that is very expensive. Logs are also poor at "is this normal?" because a single line has no sense of normal.

Pillar 2: metrics, "is something wrong, and how bad?"

A metric is a number that you keep over time. Instead of one line per request, the program keeps a few counters and updates them. A monitoring system collects them every few seconds. I ran a real Prometheus server against my checkout service and asked it three questions. These are real answers after about 7,000 requests:

requests per second (last 20s):     210.5error ratio overall:                0.0171      (1.7 percent)p99 latency (99th percentile):      0.8945 sp50 latency (the median):           0.0525 s

Four small numbers tell you the situation in one second. Traffic is about 210 requests a second, 1.7 percent fail, the typical request takes 52 ms, but 1 request in 100 takes almost 900 ms. A dashboard can draw these as lines, and an alert can watch them. Metrics are the best pillar for "is something wrong, and how bad is it?"

But a metric cannot tell you which user was slow or which SKU failed. The detail was thrown away when the numbers were added up. That is not a bug. It is the reason metrics are small and cheap. We will see how in lesson 2.

Pillar 3: traces, "where did the time go?"

A trace follows one request across services. Each step is a span, with a start time and a duration. Here is a real trace from two local services (a gateway calling an inventory service, which runs a fake database query). I printed it with a small script from the spans the OpenTelemetry SDK wrote:

trace fbb307353101cc69c23c041113679dc0  GET [gateway, SERVER] start +0ms dur 318ms    GET [gateway, CLIENT] start +1ms dur 305ms      GET [inventory, SERVER] start +3ms dur 307ms        db.query [inventory, INTERNAL] start +3ms dur 301ms    render [gateway, INTERNAL] start +307ms dur 10ms

Read it from the top. The whole request took 318 ms. Almost all of it (301 ms) was the db.query span inside the inventory service. The render step took only 10 ms. If someone says "checkout is slow", this picture tells you in a few seconds that the problem is the database query in the inventory service, not the gateway. A log or a metric could not show you this ordering of steps across two services so easily. Traces are the best pillar for "where did this request spend its time?"

The OpenTelemetry docs describe a trace the same way: the path a request takes through your application, built from spans, where the first span (with no parent) is the root span and every span in the same trace shares one trace_id. We will use exactly that in lesson 3.

How they work together

In a real incident you use all three in a loop:

1. A metric alert says "error rate is 1.7 percent, p99 is high". (Something is wrong.)
2. You look at a trace of a slow request. It shows the db.query span is slow. (Where.)
3. You open the logs for that trace and see the exact query and the exact error. (Why.)

For this to work the three must be connected. A log line should carry the id of the trace or request it belongs to, and a metric should be broken down by labels that match how you search logs (service, route, status). Without links you have three piles of data. With links you have a way to move from "something is wrong" to "this exact line of code" in a few clicks. Lesson 1 shows the request id link; lesson 3 shows the trace id link.

A tiny example of "known" and "unknown" questions

Let us make the difference between monitoring and observability concrete, using the checkout service.

A monitoring question: "Is the error rate above 5 percent?" You can write one line of PromQL (the Prometheus query language) for that. It is a known question.

An observability question: "The p99 got worse after 14:00. Is it all routes, or one? One server, or all? One customer plan, or everyone?" To answer this, your data must have those dimensions. If your latency metric only says http_request_duration_seconds with no route label, you cannot split by route. You would have to add the label, redeploy, and wait for the problem to come back. That is what low observability feels like: you cannot answer a new question without changing the code.

The opposite also exists. If you add a label with a value like user_id, you get too many separate series and your metrics system gets slow and expensive. I measured this on my machine. With a label status that has two values, a counter produced 2 series. With an extra label user_id and 10,000 different users, the same counter produced 10,000 series and the /metrics page grew to 521,072 bytes. So there is a trade-off, and part of the craft is choosing which dimensions go into cheap metrics, which go into logs, and which go into traces. Lesson 2 shows the full experiment.

The four golden signals

If you only have time to look at four things for a service that users call, look at these. The idea comes from the Google SRE book. I could not open the book from my machine for this lesson, so I describe the four signals from my own knowledge. Please read the original chapter on monitoring distributed systems before quoting it.

1. Latency. How long requests take. Look at successful and failed requests separately, because a fast failure can hide a slow-success problem (and the other way round).
2. Traffic. How much demand there is. For a web service this is usually requests per second.
3. Errors. The rate of requests that fail. This includes explicit failures (HTTP 500) and quiet ones (HTTP 200 with the wrong content).
4. Saturation. How full the system is. For example how close the CPU, memory, queue or connection pool is to its limit. Saturation often rises before latency does.

Here are the first three from my real run, as PromQL queries. Do not worry about the syntax yet. Lesson 2 explains each part:

traffic:  sum(rate(http_requests_total[20s]))                  -> 210.5 per seconderrors:   sum(http_requests_total{status="500"}) / sum(http_requests_total)   -> 0.0171latency:  histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[1m])))  -> 0.8945 s

For saturation the same service exposes values from prom-client's default collectors, such as nodejs_eventloop_lag_seconds (it showed 0.0018 s, about 1.8 ms, in one scrape) and nodejs_heap_size_used_bytes (about 9.6 MB). Event loop lag is a good saturation signal for Node: if it rises, the single thread is too busy (see The Event Loop).

RED and USE, two shorter recipes

Two other short recipes help you remember what to measure. Again, I describe them from my own knowledge since the original pages were blocked.

• RED is for services (things that handle requests): Rate (requests per second), Errors (failed requests per second or ratio), Duration (how long requests take). It is the golden signals without saturation, and it was suggested by Tom Wilkie.
• USE is for resources (things like CPU, disks, network cards, connection pools): Utilisation (how busy it is, as a percent of time), Saturation (how much extra work is waiting), Errors (count of error events). It was suggested by Brendan Gregg.

A quick way to remember: use RED for the things your users call, and USE for the things your service uses. For one server, RED asks "how is the service doing for its callers?" and USE asks "which machine part is the bottleneck?"

The Prometheus docs agree with the same basic idea for the most common service type. For an online-serving system (one where someone is waiting for an answer), the key metrics are how many queries were handled, how many failed, and how long they took. The docs also say to track requests in progress, and to be consistent about counting a request when it ends, since that lines up with the error and latency numbers.

Symptoms and causes

One more idea before the hands-on part. When you decide what to alert on, there is a big difference between a symptom and a cause.

• A symptom is what the user feels: "the page is slow", "checkout fails".
• A cause is why: "the database is out of connections", "disk is full".

The Prometheus alerting page says this clearly: alert on symptoms linked to user pain, keep the number of alerts small, make it easy to find the failing part, and avoid pages when there is nothing for the person to do. A symptom alert (high error rate on checkout) catches problems from any cause, including causes you never imagined. A cause alert (disk above 80 percent) only catches one cause. You still collect cause data, but you use it to diagnose, not to wake someone up. We build on this in lesson 5.

This is also a way to see the pillars: metrics on symptoms to detect, traces and logs on causes to diagnose.

What a good telemetry setup looks like (the plan for this series)

Here is the picture we will build, in the order of the lessons:

 user --> [gateway] --> [inventory] --> [database]             |              |   logs (JSON, request id)  logs (same request id)         <- lesson 1   metrics at /metrics (counter, gauge, histogram)         <- lesson 2   spans (same trace id across both services)              <- lesson 3             |        Prometheus scrapes /metrics every few seconds             |        SLO math + alert rules (burn rate)                 <- lessons 4, 5

Notice that every service writes its own telemetry, and a shared id (the request id or the trace id) ties it together. That is the whole architecture. The rest is detail.

Instrument for the questions you will ask

What should a beginner add first? Here is a short, practical order. It follows the Prometheus guidance that you should instrument everything a little, rather than a few things deeply, and that every library and subsystem should have at least some metrics.

1. At the edge of your service, add RED metrics for each route: a counter of requests by status, and a histogram of duration. This is lesson 2.
2. Log each request once, at the end, with the method, route, status, duration and a request id. Log errors with the error message and stack. This is lesson 1.
3. Add a trace that covers each outgoing call (database, HTTP, queue). Often the OpenTelemetry auto-instrumentation does this for you. This is lesson 3.
4. Expose saturation: event loop lag, memory, queue length, connection pool usage.
5. Add business events that matter: orders placed, payments failed. They tell you the thing users care about, which is not always visible from HTTP codes.

Observability has a cost

Telemetry is not free, and an honest lesson should say so.

• Performance. Logging, counting and tracing take CPU time and memory. Usually very little, but it is not zero. Tracing in particular is often sampled, meaning only a fraction of requests are recorded.
• Storage and money. Logs grow with traffic. Metrics grow with the number of label combinations. Traces grow with traffic and spans per request.
• People time. A dashboard nobody trusts, or an alert that fires every day, costs attention. We talk about this in lesson 5.
• Privacy and security. Logs often end up holding passwords, tokens or personal data by accident. Lesson 1 shows how to stop this with redaction. This also links to Security Lesson 8, where missing logging is itself listed as a risk.

A small demonstration of the cost trade-off in logs. I simulated 10,000 requests with a 1 percent failure rate. The policy: always log failures, and log only 1 in 20 successes. The real output:

{ errs: 81, keptSuccess: 474, lines: 555, bytes: 30232 }

Out of 10,000 requests, only 555 lines were written (81 errors plus 474 sampled successes) instead of 10,000. Every error is still there. We lose most of the "everything is fine" lines, which are the least valuable ones. The numbers are small because this is a toy, but the shape is the same in production: keep what is rare and important, sample what is common and boring. Metrics still count every request, so the totals stay correct. Lesson 1 explains sampling in more detail.

Try it

You can do this without writing any code. Take a service you know, or imagine a simple one: a "to-do list" API with routes GET /todos, POST /todos, DELETE /todos/:id.

1. Write the golden signals for it. For each, name one measurement.
2. A user says "my todos are slow to load". Which pillar do you open first, and why?
3. You open that first pillar and see "p99 latency for GET /todos is 3 s, p50 is 40 ms". Which pillar do you open next?
4. Someone suggests a metric todo_requests_total{user_email="..."}. What is wrong with that?
5. Write three alerts you would set. Mark each as a symptom or a cause. Which would you page a person for?
6. Which of the following are metrics, logs or traces? (a) "user 42 deleted todo 7 at 14:02:11" (b) "todos created per minute" (c) "the DELETE request spent 220 ms in the database"

Answers

1. Latency: p99 of GET /todos duration. Traffic: requests per second. Errors: share of requests with status 5xx. Saturation: event loop lag, or database connections in use as a share of the pool size.
2. A metric (latency percentiles by route), because you need to know whether it is really slow and for how many users. Do not start with logs, since one line cannot tell you "how many".
3. A trace of a slow GET /todos request. p99 much higher than p50 means a small share of requests are slow, and a trace shows which step (maybe one database query) took the time.
4. The label user_email has a huge number of possible values (high cardinality). It makes a new time series for every user, which hurts storage and query speed. It is also personal data. Put the user in the logs or as a trace attribute instead.
5. Example: "5xx ratio above 2 percent for 5 minutes" (symptom, page). "p99 latency above 1 s for 10 minutes" (symptom, page or ticket). "disk above 85 percent" (cause, ticket, since it is not user pain yet). Page for symptoms with real user impact.
6. (a) is a log (one event). (b) is a metric (a number over time). (c) is a trace (a span with a duration inside one request).

Interview questions

Key takeaways

• Monitoring answers questions you prepared. Observability is being able to answer new questions from the data your system already emits.
• Logs show one event in detail. Metrics show numbers over time. Traces show where one request spent its time. You need all three and they must be linked.
• In my real run, metrics showed 1.7 percent errors and a p99 of about 0.89 s, a trace showed a 301 ms database span, and logs showed the exact failing SKU.
• Look first at the four golden signals: latency, traffic, errors and saturation. RED is for services, USE is for resources.
• Averages hide the tail. Use percentiles.
• High-cardinality labels are dangerous on metrics. 10,000 user ids made 10,000 series from one counter.
• Alert on symptoms, diagnose with causes.
• Observability costs CPU, money, attention and privacy. Sample the boring data, keep the rare and important data.

What's next

Next: Logging That Helps, Lesson 1. We start with the oldest pillar. You will see why a printf-style line is hard to search, switch to structured JSON logs with pino, add levels, redact secrets, and follow one request across two services with a correlation id. Practise now: pick a service you know and write down the three questions you would ask first if it broke at 2 a.m., then check which pillar could answer each one.