Building an Evaluation Harness for an LLM Feature

Normal tests check that add(2, 3) returns 5. An LLM feature has no single right string, and a small prompt edit can quietly break a corner of it. Evals turn "it feels better" into numbers you can compare. This lesson builds a tiny harness in TypeScript, runs it, and shows it catching a regression.

For what LLM evals are in the interview sense, and how retrieval quality is measured, see vector databases and LLM evals and LLMs, agents and RAG. For ordinary test strategy see Testing From Zero if you need the basics. Lesson 1 showed that you also need injection tests; they fit in this same harness.

Words used in this lesson

• Eval (evaluation): a repeatable measurement of how well a feature does its job.
• Golden dataset: a set of inputs with known good answers or rules for a good answer, written or reviewed by people.
• Check (or assertion): a small piece of code that says pass or fail for one property of an output.
• Exact match: the output equals the expected value.
• Schema check: the output parses and has the expected fields and types.
• Rubric: a written scoring guide, for example "1 to 5 for tone".
• LLM judge: a model that scores another model's output using a rubric. Here a mock judge is a plain function.
• Baseline: the current version you compare against.
• Regression: a change that makes something that worked get worse.
• Gate: a rule in CI that fails the build when evals get worse.
• Confidence interval (CI): a range that likely contains the true rate. With few test cases it is wide. (Not the same CI as "continuous integration" below; I write "CI gate" for the build kind.)
• Position bias, verbosity bias, self-preference bias: judge habits explained below.

Why evals

Prompts and models change. A new version might fix 10 cases and break 3 you forgot. Without a fixed test set you find out from users. With one you find out in a pull request. Evals also help you decide between two options (a cheaper model or a longer prompt) with evidence instead of taste.

The loop:

change prompt/model/retrieval -> run golden set -> compare to baseline -> read failures -> fix -> repeat

The example feature

A support assistant reads a customer message and returns JSON: a category and a short reply. It must (1) return valid JSON, (2) pick the correct category, (3) never repeat a card number back in the reply, and (4) sound polite. The first three can be checked with code. The fourth needs a rubric.

Step 1. A golden dataset

Start small and real. Each case has an input, the expected category, and optional rules.

type Case = { id: string; category: string; input: string; expectCategory: string; mustNotMatch?: RegExp };const CARD = /\b(?:\d[ -]?){13,16}\b/;const dataset: Case[] = [  mk("b1", "billing", "I was charged twice this month"),  mk("b3", "billing", "Please refund order 1234, my card is 4111 1111 1111 1111", CARD),  mk("t2", "tech",    "I cannot log in after resetting my password"),  mk("a4", "account", "I want to close my account and my card is 5500 0000 0000 0004", CARD),  mk("s3", "shipping","Tracking number shows no updates for a week"),  // ... 20 cases total: 5 each for billing, tech, account, shipping];

Good golden sets share these habits:

• Cover categories and edge cases, not only happy paths: empty input, very long input, another language, an injection attempt, a sensitive-data case.
• Come from real traffic when you can (see sampling below), after removing personal data.
• Are versioned in git and reviewed like code. When a bug is found in production, add the failing input.
• Include a few cases the feature should refuse or escalate, not only ones it should answer.
• Keep a separate held-out set you do not tune on, or you will overfit the prompt to the test set.

Step 2. Cheap checks first

Order your scoring from cheap and certain to expensive and fuzzy.

The harness runs checks as small functions returning pass or fail with a reason:

function runChecks(c: Case, out: { raw: string }): CheckResult[] {  const res: CheckResult[] = [];  let parsed: any = null;  try { parsed = JSON.parse(out.raw); } catch { /* handled below */ }  const schemaOk = parsed && typeof parsed.category === "string" && typeof parsed.reply === "string";  res.push({ name: "schema", pass: !!schemaOk, detail: schemaOk ? undefined : "invalid JSON or shape" });  res.push({ name: "category_exact", pass: !!schemaOk && parsed.category === c.expectCategory });  if (c.mustNotMatch) {    const leaked = !!schemaOk && c.mustNotMatch.test(parsed.reply);    res.push({ name: "no_card_echo", pass: !leaked });  }  return res;}

Step 3. The mock model and the mock judge

I wrote two fake model versions. modelV1 classifies by keywords. modelV2 simulates a "harmless" prompt change that went wrong: it echoes the user's message back in the reply, mislabels refund and price messages, and breaks the JSON when the input is long and has a comma.

const modelV1: Model = (input) => {  const category = classify(input);  return { raw: JSON.stringify({ category, reply: `Thanks for contacting us about ${category}. We will help you shortly.` }) };};const modelV2: Model = (input) => {  let category = classify(input);  if (/refund|price/i.test(input)) category = "account";  const reply = `You wrote: "${input}". We will help you shortly.`;  const raw = JSON.stringify({ category, reply });  if (input.length > 45 && input.includes(",")) return { raw: raw + " (end of answer)" };  return { raw };};

The mock judge scores a reply from 1 to 5 using fixed rules. I made it imperfect on purpose so you can see judge pitfalls:

function judgeScore(reply: string): number {  let s = 3;  if (/thanks|thank you/i.test(reply)) s += 1;  if (/shortly|help/i.test(reply)) s += 0.5;  if (reply.length > 90) s += 0.5;      // a verbosity bias: longer looks better  return Math.min(5, s);}function judgePair(a: string, b: string): "A" | "B" {   // ties go to the first answer: a position bias  return judgeScore(b) > judgeScore(a) + 0.4 ? "B" : "A";}

A real judge would be a model call with a rubric prompt. Its interface is the same: input in, a score out. The harness does not care.

Step 4. Statistics for small test sets

With 20 cases, 17 passes (85%) does not mean the true rate is 85%. A Wilson score interval gives a sensible 95% range even for small samples (it behaves better than the simple textbook formula near 0% and 100%):

function wilson(pass: number, n: number, z = 1.96): [number, number] {  const p = pass / n, d = 1 + (z * z) / n;  const c = p + (z * z) / (2 * n), m = z * Math.sqrt((p * (1 - p)) / n + (z * z) / (4 * n * n));  return [(c - m) / d, (c + m) / d];}

Rules of thumb:

• The interval shrinks roughly with the square root of the number of cases. Four times as many cases halves the width.
• If the intervals of two versions overlap a lot, you cannot claim one is better from pass rates alone. Look at paired differences: which specific cases flipped.
• LLM outputs are random if temperature is above zero. Run each case several times, or fix the sampling settings, and report that.
• Do not hunt through many metrics until one looks good. Decide the main metric and the gate before the run.

Step 5. Run it and gate it

The harness runs both models over the dataset, prints a summary, groups failures by category, lists cases that flipped from pass to fail, and applies a gate. This is the real output (the exit code was 1, as intended):

== Summarybaseline v1 : pass 19/20 = 95.0%  95% CI [76.4%, 99.1%]  mean judge 4.50candidate v2: pass 17/20 = 85.0%  95% CI [64.0%, 94.8%]  mean judge 3.50== By category (candidate)billing   3/5tech      5/5account   4/5shipping  5/5== Baseline failuresa4 account failed: category_exact== Cases that flipped pass -> failb3  billing   failed: schema, category_exactb5  billing   failed: category_exact== GateCIs overlap: true  (with 20 cases, a 10.0% drop alone is not statistically clear)FAIL: pass rate dropped 10.0% (limit 5.0%); 1 safety case(s) failed (limit 0)== Judge pitfalls (mock judge)scores: short = 4.5  long = 5order (short, long) -> B | order (long, short) -> Asame two answers, swapped order, winner text: first / first

How to read it:

• Pass rate fell from 95% to 85%. The gate rule "drop above 5 points fails" fired.
• The intervals overlap (CIs overlap: true). With 20 cases, the drop alone is not statistically clear. That is why the gate also has a rule that needs no statistics: a hard limit on safety failures.
• Flipped cases are the real signal. b3 and b5 passed before and fail now. This paired view is more informative than the two overall rates.
• The judge also noticed: mean score fell from 4.50 to 3.50, because the echoing reply loses the polite words. A second, independent signal agreeing with the code checks raises my confidence.
• The baseline is not perfect. a4 fails for v1 too. The message contains 5500 0000 0000 0004, and the keyword 500 matched it as a "tech" message. That is a real bug in my fake classifier, found by the harness, and a good example of why you read baseline failures too.
• Only 1 safety failure is counted, from a4. Case b3 also echoes a card number in v2, but its JSON is broken, so the card check could not run (the warning above). The gate caught the regression anyway through other checks, but the leak count undercounts. Fix the check before trusting it.

Run it in CI

A gate is just a program that exits non-zero. In a pipeline:

# example step (not run here)- run: node evals/harness.ts  # exit code 1 -> the pull request cannot merge

Practical tips:

• Run a small fast suite on every pull request and a bigger suite nightly.
• Cache or record model outputs for unchanged inputs so you only pay for what changed.
• Store the baseline results as a file in the repo (or artifact) so the comparison is reproducible.
• Print the failing cases in the CI log. A red build without detail is ignored.
• Never auto-update the baseline to make the build green. A person reviews and approves a new baseline.

Judge pitfalls

An LLM judge is a model, so it has habits. The paper by Zheng et al. (arXiv 2306.05685) discusses these in the context of using a strong model as judge. From its abstract (as summarized by search; I did not re-read the full text):

The paper also lists limited reasoning ability as a limitation, and reports that some biases can be reduced. In my mock judge the same habits show up by design. This ran:

scores: short = 4.5  long = 5order (short, long) -> B | order (long, short) -> Asame two answers, swapped order, winner text: first / first

Meaning: the longer, wordier answer won in both orders, which is verbosity bias. When I gave the judge two identical answers, it picked "first" both times, which is position bias in tie cases. In a real setup you would see subtler versions of this.

More judge habits that help in practice:

• Calibrate against people. Have humans label 30 to 100 cases, check how often the judge agrees, and read the disagreements. Re-check when you change the judge.
• Prefer specific rubrics ("states the refund window; does not promise a date") over "rate the helpfulness".
• Use binary or small-scale questions where possible. Fine 1 to 10 scores are noisy.
• Keep the judge prompt and judge version fixed and versioned. A silent judge change moves all your numbers.
• Use the judge only for what code cannot check.

Sampling production traffic and error analysis

Your golden set is a guess about users. Production tells the truth.

1. Sample real requests regularly (random, plus extra from thumbs-down, escalations, errors and long-tail inputs). Remove or mask personal data first (lesson 0, LLM02).
2. Label a batch by hand and write what went wrong.
3. Group failures into categories: wrong label, format error, missing fact, leaked data, rude tone, refused when it should answer. Count each. The biggest bucket is your next fix.
4. Add the cases to the golden set so they never regress again.

In the demo, "By category" is a first version of this: billing dropped to 3/5 while tech and shipping stayed 5/5. That points directly at the billing path of the prompt change. Slice results by anything you can: category, language, input length, customer type.

Cost control

• Run code checks first. Call the judge only on outputs that passed the cheap checks, or on a sample.
• Do not judge unchanged outputs again. Key a cache on (input, model version, prompt version).
• Use a small fast suite per pull request and the full suite nightly or before release.
• Budget the eval run like any job: count calls, cap tokens, set a maximum spend that fails the job loudly. See lesson 0, LLM10. I name no prices here: check the provider's current documentation for them.
• Repeat runs only where randomness matters. For a deterministic-ish classification, one run per case may be enough.

Try it

1. Add a case. Add a golden case for a message in another language. Which check type is easiest to write? Answer: An exact-match check on the category, plus a schema check. Add a translation of an existing case so the expected category is known.
2. Why did the gate not rely only on the CI? Answer: With 20 cases the intervals overlap even for a 10-point drop. Hard rules (safety failures must be 0) and paired flips give clear signals without needing statistical significance.
3. Fix the hole in runChecks. Answer: If schemaOk is false, run c.mustNotMatch.test(out.raw) on the raw text and fail the card check when it matches, or mark every check failed.
4. Reduce position bias. Answer: Call judgePair(a, b) and judgePair(b, a). Count a win only when the answer wins in both orders; otherwise call it a tie.

Interview questions

Key takeaways

• Evals are tests for a feature whose output varies. Build a golden dataset, version it, and grow it from production failures.
• Score with cheap code checks first (exact, regex, schema). Use a judge only where code cannot, and calibrate it with human labels.
• Judges have biases: position, verbosity and self-preference. Swap order, control length, and use a different model family.
• With small sets, report intervals, read paired flips, and add hard rules for critical cases.
• Gate in CI: exit non-zero, print failures, keep the baseline in the repo.
• Slice failures by category to see what to fix next, and keep eval cost bounded with caching and cheap checks first.
• My demo used fake model and judge parts, so its numbers only show how the harness works.

What's next

Lesson 3: Cost, latency and caching uses the same idea of measuring before you optimize, applied to the money and time each request takes.