Observability and Guardrails for LLM Features in Production

Lesson 3 was about money and speed. This lesson is about keeping the feature safe and visible once real users arrive. You will see what to log per request and what never to log, how to trace a multi-step call with spans, how to use feedback and sampled checks, which guardrail layers to add, and how to handle a bad day. I build a small pipeline that does all of it, with a fake model, and run it.

Related lessons: LLM security, Lesson 0 and prompt injection, Lesson 1 explain why guardrails matter; evaluation, Lesson 2 shows how to measure quality; for general monitoring ideas (logs, metrics, traces, alerts) see Full-Stack Lesson 26 and Full-Stack Lesson 35 for tracing in short form.

Words used in this lesson

• Observability means being able to understand what your system did from what it recorded: logs, metrics and traces.
• Trace is the record of one request across all its steps. Span is one step in it (for example one model call) with a name, a duration and a result.
• Redaction means removing or masking private data (emails, card numbers) before you store or send it.
• PII (personally identifiable information) is data that points to a real person.
• Guardrail is a check or limit around the model that stops bad input or bad output from causing harm.
• Schema validation means checking that the model's answer has the exact shape and types your code expects.
• Circuit breaker is a switch that stops calling a failing dependency for a while, so you fail fast and give it time to recover.
• Kill switch is a setting that turns a feature off at once, without a deploy.
• Fallback is the safe answer or path you use when the model cannot help.
• Prompt version is a label (such as v3) for the exact prompt text in use, so you can tell which version caused what.

What to log for every request

You cannot fix what you cannot see. For each model call, record facts about the call, not the private content:

And do not log by default:

• Raw user text and raw model output. They may hold private data. If you must keep samples for debugging, keep a small, redacted, access-controlled sample with a short retention time and tell users in your privacy notice. (Check your legal rules, which differ by country.)
• Secrets, API keys, auth headers, full tool arguments that may carry tokens.
• Anything you would not want in a screenshot of your log viewer.

Redaction is a layer, not a promise

Redaction with patterns removes the obvious. It misses names, street addresses and free text such as "my daughter Anna's school is...". Treat it as one layer: redact before you send text to third parties and before you log, and limit what you collect, and control who can read the logs. The demo masks emails, card-like numbers and phone-like numbers:

redact("Mail me at ana@example.com or call +44 7700 900123, card 4111 1111 1111 1111")// -> Mail me at [EMAIL] or call [PHONE], card [CARD]

Traces and spans

A single feature call can make several steps: redact, call the model, parse the answer, maybe retry, call a tool. A trace with one span per step shows where time goes and where it fails. In the demo, each step adds a span with a name, whether it worked, and a short note. A real system would send spans to a tracing backend (OpenTelemetry is the common standard; see the note above about not naming attributes). The point is the same: one request, many labelled steps.

Feedback and online checks

• User feedback. A thumbs up/down or "this did not help" button is cheap and gives real signal. Log it with the request ID so you can find the matching trace. Be careful: feedback is biased towards people who bother to click.
• Sampled online evaluation. Take a small random sample of live requests (after redaction and with the right to do so), run the same checks you use offline (Lesson 2), and watch the trend. If a prompt change or a provider update makes the pass rate drop, you see it in days, not months.
• Cheap automatic signals: schema failure rate, retry rate, empty answers, answers that repeat the prompt, refusal rate, unusually long answers, tool errors.
• Alert on the outcome rate, not on every single error. For example, alert when "fallback" exceeds 5% of requests for ten minutes (choose your own numbers).

The guardrail layers

Think of the model as an unreliable, powerful component. Put layers around it. Each layer is simple on its own:

1. Input checks. Maximum length, allowed languages or formats, rate limits per user, and blocking of obvious abuse. (A length check stops a huge input that would cost a lot and could hide instructions.)
2. Redaction of private data before it leaves your system.
3. Prompt with a version. Keep prompts in version control or a config store, with a version label that goes into every log line. Deploy prompt changes like code: review, test with your evals, roll out gradually.
4. Timeout. A per-attempt limit and a total limit, as in Lesson 3. Do not retry a call that timed out the same way as one that returned garbage: a slow provider gets slower under retries.
5. Output validation. Parse the answer and check its shape and values. If it fails, retry a small number of times (maybe with an error message added to the prompt), then fall back.
6. Content checks. Depending on your product: a moderation service from your provider or another vendor as an extra layer, checks for leaked secrets, allow-lists for links, and removal of anything that looks like instructions to your system. These are extra layers, not guarantees (see Lesson 1).
7. Permissions on tools. The strongest guardrail is that the model cannot do harm because it was never given the power (least privilege, human approval for risky actions).
8. Fallback. A safe default response or a handover to a person.
9. Circuit breaker. After repeated failures, stop calling for a while.
10. Kill switch. A setting you can flip in seconds to turn the feature off or to switch to the fallback.

Build it: a guarded pipeline with a fake model

This program wires redaction, a prompt version, a timeout, schema validation with up to three attempts, a circuit breaker, a kill switch and a log line per request. It then runs five scenarios.

// A small guarded LLM pipeline. The "model" is a deterministic fake: no network, no API key.type Span = { name: string; ms: number; ok: boolean; note?: string };type Reply = { text: string; inTokens: number; outTokens: number };// ---- fake model: scripted answers keyed by call number, so runs are repeatableclass FakeModel {  calls = 0;  script: (n: number, prompt: string) => string | Error;  constructor(script: (n: number, prompt: string) => string | Error) { this.script = script; }  async complete(prompt: string): Promise<Reply> {    this.calls++;    const out = this.script(this.calls, prompt);    if (out instanceof Error) throw out;    return { text: out, inTokens: prompt.split(/\s+/).length, outTokens: out.split(/\s+/).length };  }}// ---- 1. redaction: remove obvious personal data BEFORE logging or sendingconst RULES: [string, RegExp][] = [  ["EMAIL", /[\w.+-]+@[\w-]+\.[\w.-]+/g],  ["CARD", /\b(?:\d[ -]?){13,16}\b/g],  ["PHONE", /\+?\d[\d -]{8,}\d/g],];export function redact(s: string): string {  let out = s;  for (const [label, re] of RULES) out = out.replace(re, `[${label}]`);  return out;}// ---- 2. output schema check (hand-written, no library)type Answer = { answer: string; confidence: number };function parseAnswer(text: string): Answer {  const v = JSON.parse(text);  if (typeof v?.answer !== "string" || v.answer.length === 0) throw new Error("answer must be a non-empty string");  if (typeof v?.confidence !== "number" || v.confidence < 0 || v.confidence > 1) throw new Error("confidence must be 0..1");  return v;}// ---- 3. circuit breaker: after N failures in a row, stop calling for a whileclass Breaker {  fails = 0; openUntil = 0;  max: number; coolMs: number; now: () => number;  constructor(max: number, coolMs: number, now: () => number) { this.max = max; this.coolMs = coolMs; this.now = now; }  get open() { return this.now() < this.openUntil; }  ok() { this.fails = 0; }  fail() { if (++this.fails >= this.max) this.openUntil = this.now() + this.coolMs; }}// ---- 4. kill switch + prompt version come from config you can change without a deployconst config = { killSwitch: false, promptVersion: "v3" };let clock = 0; const now = () => clock;const breaker = new Breaker(3, 60_000, now);const logLines: object[] = [];async function withTimeout<T>(p: Promise<T>, ms: number): Promise<T> {  let t: NodeJS.Timeout;  const timeout = new Promise<never>((_, rej) => { t = setTimeout(() => rej(new Error("timeout")), ms); });  try { return await Promise.race([p, timeout]); } finally { clearTimeout(t!); }}export async function answer(model: FakeModel, requestId: string, userText: string) {  const spans: Span[] = [];  const t0 = performance.now();  const record = (name: string, ok: boolean, note?: string) => spans.push({ name, ms: +(performance.now() - t0).toFixed(1), ok, note });  let outcome = "ok"; let result: Answer | null = null; let tokens = { in: 0, out: 0 }; let attempts = 0;  if (config.killSwitch) { outcome = "killed"; }  else if (breaker.open) { outcome = "breaker_open"; }  else if (userText.length > 2000) { outcome = "input_too_long"; }  else {    const safe = redact(userText);    record("redact", true, safe === userText ? "nothing found" : "personal data replaced");    const prompt = `[${config.promptVersion}] Answer as JSON {answer, confidence}. Question: ${safe}`;    for (let i = 1; i <= 3; i++) {      attempts = i;      try {        const r = await withTimeout(model.complete(prompt), 50);        tokens = { in: tokens.in + r.inTokens, out: tokens.out + r.outTokens };        record(`model#${i}`, true);        try { result = parseAnswer(r.text); record(`schema#${i}`, true); }        catch (e) { record(`schema#${i}`, false, (e as Error).message.slice(0, 40)); throw e; }        breaker.ok(); outcome = "ok"; break;      } catch (e) {        if ((e as Error).message === "timeout") record(`model#${i}`, false, "timeout");        breaker.fail(); outcome = "failed";        if ((e as Error).message === "timeout") break;   // do not retry a slow model      }    }    if (!result) result = { answer: "Sorry, I could not answer that. A person will follow up.", confidence: 0 }; // fallback  }  // the log line: ids, version, counts, outcome. NO raw user text.  logLines.push({ requestId, promptVersion: config.promptVersion, outcome, attempts, tokens, spans: spans.map((s) => `${s.name}:${s.ok ? "ok" : "FAIL"}${s.note ? "(" + s.note + ")" : ""}`) });  return result;}// ---- demo ----const good = JSON.stringify({ answer: "Refunds take 5 days.", confidence: 0.8 });const fakeSlow = (): Promise<Reply> => new Promise((r) => setTimeout(() => r({ text: good, inTokens: 5, outTokens: 5 }), 500));console.log("redact():", redact("Mail me at ana@example.com or call +44 7700 900123, card 4111 1111 1111 1111"));// A. a model that answers badly twice, then well (retry with schema check)const m1 = new FakeModel((n) => (n === 1 ? "Sure! Here you go." : n === 2 ? '{"answer":"x","confidence":7}' : good));await answer(m1, "req-1", "How long do refunds take? my email is ana@example.com");// B. three requests in a row that all fail -> breaker opensconst m2 = new FakeModel(() => "not json");await answer(m2, "req-2", "q"); // 3 failed attempts: breaker opensawait answer(m2, "req-3", "q"); // breaker open: model not calledconsole.log("model calls after breaker opened:", m2.calls);// C. cool-down passes, model recoversclock += 61_000;const m3 = new FakeModel(() => good);await answer(m3, "req-4", "q");// D. slow model -> timeout, no retryconst m4 = new FakeModel(() => new Error("never used"));m4.complete = fakeSlow;await answer(m4, "req-5", "q");// E. kill switchconfig.killSwitch = true;await answer(m3, "req-6", "q");config.killSwitch = false;for (const l of logLines) console.log(JSON.stringify(l));console.log("\nraw email present in any log line?", JSON.stringify(logLines).includes("ana@example.com"));

Real output from node pipeline.ts:

redact(): Mail me at [EMAIL] or call [PHONE], card [CARD]model calls after breaker opened: 3{"requestId":"req-1","promptVersion":"v3","outcome":"ok","attempts":3,"tokens":{"in":48,"out":9},"spans":["redact:ok(personal data replaced)","model#1:ok","schema#1:FAIL(Unexpected token 'S', \"Sure! Here you go)","model#2:ok","schema#2:FAIL(confidence must be 0..1)","model#3:ok","schema#3:ok"]}{"requestId":"req-2","promptVersion":"v3","outcome":"failed","attempts":3,"tokens":{"in":24,"out":6},"spans":["redact:ok(nothing found)","model#1:ok","schema#1:FAIL(Unexpected token 'o', \"not json\" is not )","model#2:ok","schema#2:FAIL(Unexpected token 'o', \"not json\" is not )","model#3:ok","schema#3:FAIL(Unexpected token 'o', \"not json\" is not )"]}{"requestId":"req-3","promptVersion":"v3","outcome":"breaker_open","attempts":0,"tokens":{"in":0,"out":0},"spans":[]}{"requestId":"req-4","promptVersion":"v3","outcome":"ok","attempts":1,"tokens":{"in":8,"out":4},"spans":["redact:ok(nothing found)","model#1:ok","schema#1:ok"]}{"requestId":"req-5","promptVersion":"v3","outcome":"failed","attempts":1,"tokens":{"in":0,"out":0},"spans":["redact:ok(nothing found)","model#1:FAIL(timeout)"]}{"requestId":"req-6","promptVersion":"v3","outcome":"killed","attempts":0,"tokens":{"in":0,"out":0},"spans":[]}raw email present in any log line? false

How to read it:

• req-1: the first answer was not JSON (schema#1 failed), the second had confidence 7 (failed the range check), the third was valid. It cost 3 attempts, and the log line has the token totals. The email in the question is not in the log (see the last line: false).
• req-2: three bad answers in a row. The call ends with the fallback text and the breaker counts three failures, so it opens.
• req-3: the breaker is open, so the model is not called at all (attempts: 0; the script also printed that the model had been called 3 times in total). The user still gets the fallback.
• req-4: after the cool-down the breaker allows a call, and it succeeds.
• req-5: a slow model hits the timeout and is not retried.
• req-6: the kill switch is on, so there is no model call and no cost.

The timings are not shown because the demo's value is the decisions. Notice that every path produces a log line with the same fields, so a dashboard can count outcomes: ok, failed, breaker_open, killed.

Prompt versions and rollback

Treat the prompt as code:

1. Keep it in the repository (or a config store) with a version label.
2. Put the version in every log line (the demo does).
3. Before you ship a new version, run your evaluation set (Lesson 2) and compare with the old one.
4. Roll out to a small share of traffic first and watch the outcome rates.
5. Keep the old version ready. Rolling back should be a config change, not a deploy.

The same applies to the model you use: log the model identifier your code sends, and record when you change it. A provider changing a model behind the same name can change your results, which is one more reason to run sampled checks.

An incident playbook for a bad model output

When a user shows you a harmful, wrong or embarrassing answer:

1. Stop the harm. Flip the kill switch or switch to the fallback for the affected feature. Do this first, then investigate.
2. Find the request with the request ID or the time, and the prompt version, model, inputs and tool calls (this is why you logged them).
3. Classify the cause: prompt flaw, missing input check, injection from untrusted content (Lesson 1), data problem in retrieval, a provider change, or a bug in your validation.
4. Fix and test. Add the failing case to your evaluation set so it can never come back unnoticed.
5. Tell people who were affected, if your rules or policy require it, and follow your security process if private data was exposed.
6. Write a short blameless review (Team Lead, Lesson 4): what happened, what the system allowed, what we change.

Try it

1. Add a rule to RULES for a national ID number format you know and show it being masked. What can still leak (think of names)?
2. Change the breaker to open after 2 failures. Which of the five scenarios change?
3. Add a fallbackCount metric and print the share of requests that ended with the fallback.
4. Add a rule that blocks any model answer containing the text http:// and decide what the user should see.

Answers. (1) Add a regular expression for the format and a test. Names, addresses and free-text details can still leak, which is why redaction is only one layer. (2) My loop does not check breaker.open between attempts, so req-2 still makes three attempts; the breaker simply opens after the second failure and req-3 is blocked as before. To stop the third attempt you would add a breaker.open check inside the loop. (I reasoned this from the code and did not run the change.) (3) Count entries whose outcome is not ok, or where the fallback text was used, and divide by the total. (4) Check the string after parsing, replace the answer with the fallback, log outcome: "blocked_output", and show the safe fallback message.

Interview questions

Key takeaways

• Log facts about every call (IDs, prompt version, model, tokens, latency, attempts, outcome), not private content. Redaction is one layer, not a guarantee.
• Use traces with one span per step to see where time goes and what fails.
• Add layers: input limits, redaction, versioned prompts, timeouts, output validation, content checks, least-privilege tools, fallback, circuit breaker, kill switch.
• Every code path should end in one structured log line and a safe response.
• Treat prompts like code: version, test with evals, roll out gradually, roll back by config.
• Watch outcome rates and sampled quality, because models and providers change.
• In an incident, stop the harm first, then investigate, then add the case to your evals.

What's next

You have finished the AI engineering series. Good next steps are the interview-style lessons LLMs, agents and RAG, vector databases and LLM evals, and the testing ideas in Testing From Zero. To practise, take any small LLM feature you have and write down: what you log, your fallback, and your kill switch.