Logs, Metrics & Traces: The Three Pillars

1. Logs: detailed records of events

A log is a timestamped record of something that happened: a request, an error, a decision.

Unstructured logs are plain sentences. People can read them, but machines find them hard to search:

2027-03-08 10:42:13 ERROR Payment failed for user 42 order 5521 after 3012ms: card declined

Structured logs put each fact in its own named field. They are usually written as JSON (a text format made of key-value pairs). They are much better:

{  "timestamp": "2027-03-08T10:42:13.482Z",  "level": "error",  "service": "payment-service",  "message": "payment failed",  "user_id": 42,  "order_id": "5521",  "duration_ms": 3012,  "error_code": "card_declined",  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",  "version": "2027.03.08-1"}

Now you can search and filter by any field ("all error_code=card_declined in the last hour, grouped by version").

Logging best practices:

• Use log levels: DEBUG (development detail), INFO (normal events), WARN (unusual but handled), ERROR (something failed), FATAL (the service is crashing). In production, usually log at INFO and above.
• Include context: request ID, trace ID, user or tenant ID, service name, version, and duration.
• Use correlation IDs. A correlation ID is a unique label that you attach to one request. Create a request ID at the edge (the first server that receives the request). Pass it to every service in an HTTP header (extra information sent with a request). Then you can find all log lines for one request across the whole system.
• Centralise logs. Ship logs from every container to one searchable place: the ELK/Elastic stack, OpenSearch, Grafana Loki, Splunk, Datadog or cloud logging services. Local log files on containers disappear when the container does.
• Never log secrets or sensitive data: passwords, tokens, full card numbers, OTPs. Be careful with personal data (Part 9). Mask or remove it.
• Watch the cost. Logging everything at high traffic can cost more than the servers themselves. Sample the noisy success logs (keep only a small share, for example 1 in 100) and keep all errors. Also set a retention period, which is how long you keep data. For example, keep 7–30 days of searchable logs, and move older ones to cheap object storage.

2. Metrics: numbers over time

A metric is a number that you measure and record again and again over time, such as "requests per second". It often has labels (also called dimensions or tags). A label is a small name-value tag, such as region=eu, that lets you split the number into groups.

Metric types:

• Counter: only goes up (total requests, total errors). You usually look at its rate ("errors per second").
• Gauge: goes up and down (memory used, queue length, active connections, temperature).
• Histogram: counts values in buckets (for example, how many requests took under 50 ms, under 100 ms, under 250 ms). From the buckets you can work out percentiles (post 8). A percentile is the value below which a given share of requests fall. For example, p95 = 200 ms means 95% of requests were faster than 200 ms. Use histograms for latency, not averages.
• Summary: works out percentiles inside each service (on the client side). It is simpler, but you can't combine the percentiles of many servers into one correct overall percentile. So histograms are usually preferred.

Why metrics are powerful:

• Cheap to store: a number every 10–60 seconds, no matter how much traffic there is.
• Fast to query over long periods ("show error rate for the last 30 days").
• Perfect for dashboards and alerts.

The cardinality trap. Cardinality means how many different values something can have. A time series is one list of numbers over time. Every unique combination of labels creates its own separate time series:

Never use unbounded values (user IDs, order IDs, email addresses, full URLs with IDs) as metric labels. Put those in logs and traces instead.

Pull vs push:

• Pull: the metrics system scrapes each service. This means it reads (downloads) the current numbers from the service's /metrics endpoint every few seconds. Prometheus (a popular open-source metrics system) works this way.
• Push: services send metrics to a collector (StatsD, OpenTelemetry OTLP, many SaaS agents).

What should you measure? Three proven frameworks

The Four Golden Signals come from Google's SRE book. SRE (Site Reliability Engineering) is the way Google runs services reliably. The four signals apply to any service that users call:

1. Latency: how long requests take. Track the time of failed requests separately.
2. Traffic: how much demand (requests per second).
3. Errors: the rate of failed requests.
4. Saturation: how "full" the service is (CPU, memory, queue depth, connection pool usage).

The RED method is for services that answer requests: Rate (requests per second), Errors (how many fail) and Duration (how long they take).

The USE method (from Brendan Gregg) is for resources like CPU, disks and network: Utilisation (how busy), Saturation (how much work is waiting), Errors.

A good default: RED for every service, USE for every resource, and business metrics like orders per minute, sign-ups and payment success rate. Business metrics often catch problems that technical metrics miss.

3. Traces: following one request across services

A distributed trace records the full path of one request through all the services it touches, with the time of each step. "Distributed" means the request travels across many services.

• A trace is the whole journey, identified by a trace ID.
• A span is one step in that journey (one service call, one database query), with a start time, a duration, attributes and a parent span.

Trace 4bf92f35…  POST /checkout  (total 3,240 ms)
│
├─ api-gateway                      [■■                                    ]   12 ms
├─ cart-service: get cart           [ ■■■                                  ]   38 ms
│   └─ redis GET cart:42            [ ■                                    ]    2 ms
├─ pricing-service: calculate       [    ■■■■                              ]   61 ms
│   └─ postgres SELECT prices       [     ■■■                              ]   44 ms
├─ inventory-service: reserve       [        ■■■                           ]   40 ms
└─ payment-service: charge          [           ■■■■■■■■■■■■■■■■■■■■■■■■■■■] 3,012 ms  ← here!
    └─ HTTP POST payment-provider   [            ■■■■■■■■■■■■■■■■■■■■■■■■■■] 2,998 ms

One glance shows that the payment provider is the slow part, not your code.

How it works: context propagation. Context propagation means passing the trace information from one service to the next. When service A calls service B, it sends the trace ID and its own span ID in an HTTP header. The W3C Trace Context standard (a shared web standard) defines this header:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01             version-trace_id-parent_span_id-flags

Service B continues the same trace and creates child spans. You must also pass the context through queues and events by putting it in the message headers. If you do not, the trace breaks wherever work is done later (asynchronously, Part 6).

Sampling. Sampling means recording only some requests instead of all of them. Recording every span of every request is expensive at high traffic. You have two options:

• Head sampling: decide at the start (for example, keep 1% of traces). Simple, but you might miss rare errors.
• Tail sampling: hold the spans for a short time, then decide after the request has finished. Keep all slow or failed traces plus a small share of normal ones. This is more useful, but it needs more infrastructure.

Tools: Jaeger, Zipkin, Grafana Tempo, and commercial tools like Datadog, Honeycomb, New Relic and Dynatrace.

OpenTelemetry: one standard for all three

OpenTelemetry (OTel) is an open standard and a set of tools for producing and collecting traces, metrics and logs. "Vendor-neutral" means it is not tied to one company's product. It is a project of the Cloud Native Computing Foundation (CNCF), a non-profit that hosts open-source cloud projects.

• SDKs and auto-instrumentation for most languages. Instrumentation means adding code that records traces, metrics and logs. With auto-instrumentation, many frameworks, HTTP clients and database drivers record data without you writing that code.
• The OTel Collector receives data, processes it (samples it, filters it, removes sensitive fields) and exports it to the storage tools (backends) you choose. Services send data to it with OTLP, the OpenTelemetry protocol.
• Standard names ("semantic conventions") for common attributes (extra details attached to data), like http.route and db.system.

If you instrument once with OpenTelemetry, you can switch vendors without rewriting your code.

Connecting the three pillars

The real power comes from linking them:

1. An alert fires: "checkout p99 latency > 2 s" (metric).
2. You open the latency graph and click on an exemplar, a sample trace attached to a histogram bucket, to jump to a slow trace.
3. The trace shows the payment span is slow. You click it to see its logs (they share the trace_id), which say "provider timeout, retrying".

That path from metric → trace → logs takes minutes instead of hours, if you include trace IDs in logs and use consistent service names and labels everywhere.

Monitoring vs observability

• Monitoring answers known questions, such as "Is the error rate above 1%?" or "Is the disk full?" You decide in advance what to watch.
• Observability is the ability to answer new questions you didn't plan for: "Why are checkouts slow only for Android users on app version 5.2 in Pune paying with a specific bank?"

Good observability needs rich, high-cardinality context. High cardinality means a field can have very many different values, like user ID or device. You attach user, device, version, region and similar fields to events and traces. Then you can group and filter the data in any way during an investigation. Tools like Honeycomb made this "wide events" approach popular. A wide event is one record that carries many fields.