Logging and Observability: Structured Logs, Correlation IDs, Metrics

On the frontend, you can reproduce a bug in DevTools. On the backend, you often cannot. It happened to someone else, at 2 a.m., once. "Observability" means being able to find out what happened inside your system, even when you were not there. Plan it before you need it.

Words used in this lesson

• A log is a record, written by your program, of something that happened. Logging is the act of writing these records. Example: "order 42 created".
• Observability is how well you can understand what happens inside a running system by looking at the data it produces (logs, metrics and traces).
• A metric is a number measured over time, such as requests per second. Metrics are cheap to store and good for graphs and alerts.
• A trace is the record of one request as it travels through the parts of a system. It is made of spans. A span is one timed step, such as one database call.
• A request ID (or correlation ID) is a unique value given to one request and put on every log line for it. It lets you join all the lines of that request.
• A log level is a label that says how serious a line is: trace, debug, info, warn, error, fatal (from least to most serious).
• Structured logging means writing each log line as data (usually one JSON object) with named fields, not as free text.
• JSON is a text format for data, made of {"key": value} pairs.
• stdout / stderr are the two standard output streams of a program. By default, stdout is for normal output and stderr is for errors. Platforms read them to collect logs.
• Redaction means replacing secret values in a log with a placeholder such as [REDACTED] before the line is written.
• pino is a fast Node.js logging library that writes JSON lines.
• Latency is the time between sending a request and getting the answer.
• A percentile (p50, p95, p99) is the value below which that share of measurements fall. If p95 latency is 300 ms, then 95 out of 100 requests took 300 ms or less. p50 is the median.
• The event loop is the part of Node.js that takes the next piece of waiting work and runs it. Event-loop lag is how late it is in starting that work. High lag means your code is blocking it.
• Cardinality is the number of different values a label or field can take. A status label has low cardinality (a few values). A userId label has high cardinality (millions of values).
• An SLO (service level objective) is a target for service quality, for example "99.9% of requests succeed". An error budget is the amount of failure the SLO still allows.
• A runbook is a written guide that says what to do when a given alert fires.
• Sampling means keeping only a share of the events (for example 1 in 10) to save cost.
• Retention is how long stored data is kept before it is deleted.
• A tenant is one customer (or organisation) in a system that serves many customers.
• A commit SHA is the unique ID of one version of your code in Git. Putting it in logs tells you which release wrote the line.

Difference in one line: a log says what happened once, a metric counts it over time, and a trace shows where one request spent its time.

Why console.log is not enough

Structured logs

A structured log is not a sentence. It is data with named fields, so a tool can search it. In pino, a level is also stored as a number (trace 10, debug 20, info 30, warn 40, error 50, fatal 60). That is why the example below shows "level":30, and why you can filter with level>=50.

import pino from "pino";export const logger = pino({  level: process.env.LOG_LEVEL ?? "info",  redact: {    paths: [      "req.headers.authorization", "req.headers.cookie",      "password", "token", "refreshToken", "cardNumber",      // at the top level of the logged object      "*.password", "*.token", "*.refreshToken", "*.cardNumber", // one level down, such as { user: { password } }    ],    censor: "[REDACTED]",  },  base: { service: "orders-api", env: process.env.NODE_ENV },  timestamp: pino.stdTimeFunctions.isoTime,});logger.info({ orderId: "ord_42", userId: "u_9", durationMs: 87, itemCount: 3 }, "order created");

The output is one JSON object per line:

{"level":30,"time":"2026-05-01T09:30:00.123Z","service":"orders-api","env":"production","orderId":"ord_42","userId":"u_9","durationMs":87,"itemCount":3,"msg":"order created"}

A redact path matches the field names in the object you log. * matches one level only, so list top-level names and nested names on purpose. Now you can ask a question like this: "show all level>=50 for service=orders-api where durationMs > 1000 in the last hour". You can ask it in CloudWatch Logs Insights, Loki, Datadog or Elasticsearch. You do not need to search through text.

Log levels

Make the level changeable at run time. Use an environment variable, and ideally allow a change without a new deploy. Then you can turn on debug for one instance during an incident.

What a good log line contains

• What happened: a stable message such as "order created". Do not write a sentence with values inside it.
• Who or what it was about: userId, orderId, tenantId as fields.
• Correlation: requestId and, if you use tracing, traceId.
• Result and cost: status, durationMs, retry count.
• Where: service, version or commit SHA, environment, hostname.
• For errors: the error object (message, stack, cause, error code).

Keep the message the same every time and put the changing values in fields. Fixed messages are easy to group and count. Messages with values inside are not.

What you must never log

Never log secrets or sensitive personal data. This includes passwords, tokens, API keys, session IDs, full card numbers, CVV, government IDs, health data, and full request bodies of sensitive endpoints. OWASP's logging guidance and privacy laws (GDPR, India's DPDP Act) both apply here. Use redaction by field path. Log IDs instead of the data. Cut large payloads short. Also be careful with whole req.headers (they hold cookies and auth) and with whole error responses from third parties.

Correlation IDs: tracing one request across everything

Give each incoming request an ID. Add it to every log line made while handling that request. Send it back to the client. Pass it on to the calls you make to other services.

import { randomUUID } from "node:crypto";import { AsyncLocalStorage } from "node:async_hooks";import type { RequestHandler } from "express";const als = new AsyncLocalStorage<{ requestId: string; userId?: string }>();const SAFE_ID = /^[\w-]{8,64}$/; // letters, digits, "_" and "-" onlyexport const requestContext: RequestHandler = (req, res, next) => {  const incoming = req.header("x-request-id");  const requestId = incoming && SAFE_ID.test(incoming) ? incoming : randomUUID(); // honour a safe upstream ID  res.setHeader("x-request-id", requestId);  als.run({ requestId }, next);};// A child logger that automatically includes the context, so you do not pass IDs around by handexport const log = () => logger.child(als.getStore() ?? {});

AsyncLocalStorage is a Node.js class that keeps a value attached to the current chain of asynchronous calls. Here it keeps the data of the current request, like a note that follows the request around. Accept an upstream ID only if it looks safe (the SAFE_ID check above). A client could otherwise send a huge or odd value that ends up in every log line. A child logger (logger.child(...)) is a logger that adds fixed fields to every line it writes. Deeper code (a service, a repository, a helper) can call log().info(...), and the requestId is added for you. This works even across await calls.

For systems with many services, use the W3C traceparent header. It is an HTTP header, defined by a W3C standard, that carries the trace ID from one service to the next. OpenTelemetry is an open-source toolkit (libraries and a data format) for creating traces, metrics and logs. It instruments Node, Express, Prisma and fetch. It sends traces and metrics to a backend (Jaeger, Tempo, Datadog, AWS X-Ray and others).

The access log

An access log is a record with one line for each request that the server handled. One structured line for each request is the cheapest and most useful signal:

// Register it after requestContext: app.use(requestContext, accessLog)export const accessLog: RequestHandler = (req, res, next) => {  const start = process.hrtime.bigint();  const reqLog = log(); // take the logger now, while the request context is active  res.on("finish", () => {    reqLog.info({      method: req.method,      route: req.baseUrl + (req.route?.path ?? "") || "unmatched", // pattern, not the raw URL (low cardinality)      status: res.statusCode,      durationMs: Number(process.hrtime.bigint() - start) / 1e6,      bytes: res.getHeader("content-length"),      userId: req.user?.id, // set by the auth middleware (Lesson 06)    }, "request completed");  });  next();};

Use the route pattern (/orders/:id) and not the raw path (/orders/8412). If you use the raw path, your metrics and queries split into millions of different URLs. (Having too many different values is called high cardinality.)

The three pillars

Metrics worth having (RED and USE)

• RED for every service or endpoint: Rate (requests per second), Errors (the share that fail, such as 5xx), Duration (latency, shown as p50/p95/p99, not averages).
• USE is a method for resources (CPU, memory, DB connections, disk). It checks: Utilisation (how busy), Saturation (how much work is waiting), Errors.
• Node-specific: event-loop lag (measured with perf_hooks.monitorEventLoopDelay), heap used (the memory your objects take), GC pauses (GC means garbage collection: the engine frees memory that is no longer used, and may pause your code for a moment), and open handles (open sockets, files and timers that keep the process alive).
• Business metrics: orders per minute, signups, payment failures. They often show a problem before the technical metrics do.

Averages hide pain. Say 1% of requests take 10 seconds. The average still looks fine, but thousands of users suffer. Always look at percentiles. A p99 of 2 seconds means 99 out of 100 requests were faster than 2 seconds.

Alerting

An alert is an automatic message sent to people when a metric crosses a limit. Alert on symptoms that users feel (error rate, latency, failed checkouts). Do not alert on every possible cause (such as CPU at 70%). Wake a person only for urgent problems they can act on. Everything else becomes a ticket or a dashboard. Add a runbook link (a guide that says what to do) to each alert. (The System Design series explains SLOs and error budgets in depth.) Alert fatigue is what happens when there are so many alerts that people stop reacting to them.

Where do logs go?

• Write JSON to stdout/stderr. Do not manage log files or rotation (starting a new file when the old one gets big, and deleting old ones) inside the app. This follows the twelve-factor approach, a well-known set of rules for cloud apps, which says to treat logs as a stream of events that the platform collects.
• The platform collects the logs. Examples are CloudWatch Logs on AWS (the DevOps series covers this), Docker log drivers, or a collector program (Fluent Bit or the OpenTelemetry Collector) that reads the logs and sends them to a store such as Elasticsearch, Loki or Datadog.
• Choose retention (how long you keep logs) on purpose. Logs cost money. They may also hold personal data that you must not keep forever.
• Think about sampling (keeping only some) for logs that are high in volume and low in value, like successful health checks (calls that only ask "are you alive?"). Keep all errors.

Scenario: an intermittent 500 you cannot reproduce

Here is what you need already in place:

1. A request ID in the error response that the user can quote ("Reference: 7f3a…").
2. Logs found by that ID that show the whole path of the request. This includes the unhandled error with its stack and cause.
3. The shape of the inputs (not secrets): which endpoint, which user or tenant, key parameters, payload size.
4. Timings of dependencies (database, cache, third parties) from traces or log fields. Was it a timeout?
5. Metrics from around that time. Was there a deploy, a traffic spike, full DB connections, event-loop lag or memory pressure?
6. Version tags (commit SHA) on logs, so you know which release made the error.

Then the investigation goes like this. Find the request ID and read the story. Form a guess. Check the metrics for a link. Then add a targeted log or a regression test. Without these things, you are only guessing.

Common mistakes

1. console.log in production with strings joined together.
2. Logging secrets or full request bodies.
3. No request ID, so lines from the same request cannot be linked.
4. Logging at every layer (duplicate errors), or logging so much that the useful lines are lost.
5. Metric labels with too many values (user IDs, raw URLs). A label is a name=value tag on a metric. Each different value creates a new series to store, so the metrics store grows too big.
6. Averages instead of percentiles.
7. Alerting on causes instead of symptoms. This causes alert fatigue (people stop trusting alerts).
8. Treating logging as an afterthought, with no structured fields and no retention policy.

Interview questions

Key takeaways

• Use a structured, levelled logger that hides secrets (pino) and writes JSON to stdout.
• Every request gets a request ID. AsyncLocalStorage carries it, you return it to the client, and you pass it on to other services.
• Collect RED metrics with percentiles. Add event-loop lag and business metrics.
• Never log secrets or personal data. Keep messages fixed and put variables in fields.
• Alert on symptoms, add runbooks, and design observability before the incident.

What's next

Lesson 06: Authentication and Authorization on the Server. You can now see what your API is doing. Next: making sure only the right people can do things.