Observability & SRE cheat sheet

• Monitoring answers questions you prepared. Observability answers new questions from data the system already emits.
• Order in an incident: metric (how bad), trace (which hop), log (which exact case). Link them with ids.
• Averages hide the tail: use percentiles. Alert on symptoms, diagnose with causes.
• Observability costs CPU, money, attention and privacy. Sample the boring data, keep errors.

• JSON lines with a fixed message and variable facts in fields.
• Log errors under the err key. { e } printed as {} in the test.
• Log each failure once, where you handle it. Never log secrets. Add redact as a safety net.
• Cost was about 6 microseconds per line, near zero when the level is off.

1. Reuse the incoming request id or make one at the edge.
2. Put it on a child logger so every line carries it.
3. Send it on every outgoing call and inside queue messages.
4. Search the id across services to read the full story.

Tip: Sample the boring lines, keep all errors, take totals from metrics. JSON escapes line breaks, so users cannot fake log lines.

• Real run: average latency 81 ms, p99 about 820 ms.
• Histogram percentiles are estimates. Coarse buckets showed p95 0.10 s when the truth was 0.60 s.
• Never average percentiles. Sum the buckets across servers, then take the quantile.
• Every label combination is a series. A user_id label turned 2 series into 10,000 (521 KB /metrics).
• Safe labels: method, route template (/users/:id), status class, region, version.
• Names: base units and _total. Initialise key counters. Prometheus gives up for free; check config with promtool.

• A trace is a tree of spans with one trace id. A span has name, timing, attributes, events, status.
• Always call span.end(). Name spans by operation. Keep secrets out of attributes.
• API in libraries, SDK in apps. Start the SDK before the code it instruments. Set OTEL_SERVICE_NAME.
• Put trace_id and span_id in log lines. Queues: inject context into the message, extract in the worker.

traceparent:
 00-<trace-id>-<parent-span>-01
 ver  32 hex    16 hex      flags

Tip: Parent-based head sampling kept 10% of traces and never half a trace. Tail sampling keeps the interesting ones but needs a collector.

• SLI = good events / valid events, from the user's side. SLO = SLI + target + window. SLA = contract, looser than the SLO.
• Latency SLI = share of requests under a threshold. The histogram needs a bucket at that threshold.
• Never 100%. Rolling windows are harder to game. Dependencies in series multiply.

Tip: Three dependencies at 99.9% in series give 99.7%. Dashboard green but users unhappy: the SLI is wrong.

budget    = 1 - SLO
burn rate = error ratio / budget
threshold = fraction x
  window_hours / alert_hours

• The budget is meant to be spent on change. Agree a freeze policy with the product owner while it is calm.
• Low traffic makes budgets jumpy. Count budget in requests.

• A page must be urgent, actionable and about user pain. Otherwise: ticket, dashboard or nothing.
• Every page links a runbook. Fight fatigue: count the actionable share of pages.
• Never alert on average latency. Beware low traffic, long for:, and missing data. Add a dead man's switch.

Tip: Join two windows with and: long means big enough, short means still happening, so the alert clears on recovery. Test with promtool test rules.

• Alertmanager groups, routes, inhibits (a big alert hides its children) and silences.
• One dashboard, one question. Overview: traffic, errors, percentiles, saturation, deploy markers. Keep it in version control.

• Roles: IC (coordinates, rarely fixes), deputy, scribe, experts, liaison. One channel, regular updates.
• Mitigate first, understand later. Small reversible changes, one at a time, confirm with the SLI.
• Unsure about severity? Treat it as the higher one.

Watch out: Mean time to recover (MTTR) hides uneven incidents. Track detect, ack, mitigate, resolve, and budget used per incident.

• Ask what in the system made the mistake easy. Blameless is not unaccountable.
• List contributing factors, not one root cause.
• Timeline, impact in budget terms, what went well, what was lucky.
• A good action item is small, has an owner and a date, and is tracked.
• On-call: primary and secondary, real rotation size, handoffs, rest, improve one thing per shift.

• A slow dependency is worse than a dead one. Timeout on every call, a bit above healthy p99. Share one deadline.
• Retry only passing failures on idempotent calls (or with an idempotency key), inside a deadline, with a retry budget.
• Layers multiply: 3 x 3 x 3 = 27 calls. Retry in one layer only.

delay = random() *
  min(cap, base * 2 ** attempt)

• Breaker: one per dependency, % over a window plus a minimum volume. Count timeouts and 5xx, not 4xx. Export its state.
• Bulkhead: a separate limit per dependency, so a slow optional one cannot starve a critical one.
• Load shedding: refuse early (503/429, Retry-After). An unlimited queue just turns overload into delay.
• Degradation: plan B decided in advance, marked degraded, counted. Stale price to browse, never to charge.
• Test with fault injection: hangs, slowness, flapping, partial failure.