TDD, Property-Based Testing, Mutation Testing and a Team Strategy

This is the last lesson of the series. Lesson 0 explained why we test and the shapes of a suite. Lesson 1 taught unit tests and doubles, Lesson 2 UI components, Lesson 3 integration tests and Lesson 4 browser tests. Here we ask the harder question: how good are our tests? For the interview view of a test plan see Team Lead, Lesson 6, and for tool reference see Full-Stack Lesson 21.

Words used in this lesson

• TDD (test-driven development): writing a failing test first, then the smallest code that passes it, then cleaning up.
• Red, green, refactor: red is a failing test, green is a passing test, refactor is improving the code while staying green.
• Kata: a small practice exercise you repeat to build a skill.
• Property: a rule that must be true for all valid inputs ("sorting twice gives the same result as sorting once"), not just for one example.
• Property-based testing: a tool generates many random inputs and checks your property for each.
• Shrinking: when a random input fails, the tool tries smaller, simpler inputs that still fail, to give you a small counterexample.
• Counterexample: an input that breaks the property.
• Seed: the number that starts the random generator. The same seed replays the same inputs.
• Mutation testing: a tool makes small changes (mutants) to your code, such as < to <=, and runs your tests on each one.
• Mutant killed / survived: a mutant is killed if some test fails. It survives if all tests still pass, which means your tests did not notice the change.
• Mutation score: killed mutants divided by total mutants, as a percentage.
• Code coverage: the share of code lines or branches that ran while the tests ran.

1. TDD: red, green, refactor

The loop is:

1. Red. Write one small test for behaviour that does not exist yet. Run it. Watch it fail, for the right reason.
2. Green. Write the simplest code that makes it pass. It is allowed to be ugly or even fake.
3. Refactor. Clean up, with all tests green. Run the tests after each change.

Why watch the test fail? A test you never saw fail might not test anything. It might pass for the wrong reason (this is the bug behind many "green" suites). The red step is a check on the test itself.

TDD is a tool, not a religion. It works best for logic with clear rules (parsers, calculations, rules engines) and for fixing bugs (write the failing test that reproduces the bug first). It is less natural for exploratory UI work, where you do not yet know what you want. Many good engineers use it for some work and not for other work.

2. A real kata session: the String Calculator

The kata: write add(text) that adds numbers in a string. I did every step below and ran the tests each time. I show the failure message from every red step.

Step 1: the empty string

Test first:

// test/calc.test.jsimport { test } from 'node:test';import assert from 'node:assert/strict';import { add } from '../src/calc.js';test('empty string is 0', () => { assert.equal(add(''), 0); });

With the stub export function add(s) { throw new Error('not implemented'); } the run is red:

✖ empty string is 0 (1.037656ms)ℹ pass 0ℹ fail 1  Error: not implemented

It fails because the function is missing, which is the right reason. Now the simplest green code. It is allowed to be a fake:

export function add(s) { return 0; }

✔ empty string is 0 (0.927927ms)ℹ pass 1

Returning 0 feels silly, but it is a real technique ("fake it till you make it"): the next test forces the real code.

Step 2: one number

test('one number is itself', () => { assert.equal(add('5'), 5); });

✔ empty string is 0 (0.863523ms)✖ one number is itself (0.88333ms)    actual: 0,    expected: 5,

Red, for the right reason: we return 0. Green:

export function add(s) {  if (s === '') return 0;  return Number(s);}

ℹ pass 2ℹ fail 0

Step 3: two numbers

test('two numbers are summed', () => { assert.equal(add('1,2'), 3); });

✖ two numbers are summed (0.766056ms)    actual: NaN,    expected: 3,

Number('1,2') is NaN. Green:

export function add(s) {  if (s === '') return 0;  return s.split(',').map(Number).reduce((a, b) => a + b, 0);}

✔ two numbers are summed (0.165701ms)ℹ pass 3

Step 4: newlines as separators

test('newlines also separate numbers', () => { assert.equal(add('1\n2,3'), 6); });

✖ newlines also separate numbers (0.93811ms)    actual: NaN,    expected: 6,

Green, by changing one thing, the split pattern:

return s.split(/[,\n]/).map(Number).reduce((a, b) => a + b, 0);

ℹ pass 4ℹ fail 0

Step 5: negative numbers throw

test('negatives are rejected and listed', () => {  assert.throws(() => add('1,-2,-3'), { message: 'negatives not allowed: -2,-3' });});

✖ negatives are rejected and listed (1.784712ms)  AssertionError [ERR_ASSERTION]: Missing expected exception.    actual: undefined,    expected: { message: 'negatives not allowed: -2,-3' },

Red: the function happily returned -4. Green plus a refactor into named steps:

export function add(s) {  if (s === '') return 0;  const nums = s.split(/[,\n]/).map(Number);  const negatives = nums.filter((n) => n < 0);  if (negatives.length > 0) throw new Error(`negatives not allowed: ${negatives.join(',')}`);  return nums.reduce((a, b) => a + b, 0);}

✔ negatives are rejected and listed (2.700673ms)ℹ pass 5ℹ fail 0

What the session shows

• Each step was tiny: one new test, one small change. The failure message told me what to do next.
• The design came from the tests. I never planned "split by regex"; the third and fourth tests pulled it out.
• The list of tests reads like a specification of the function.

3. Property-based testing with fast-check

An example test says "for 1 and 2 the answer is 3". A property says "for any list of numbers, the answer is their sum". A tool, here fast-check, generates many inputs (100 by default) and tries to break the rule.

// test/props.test.jsimport fc from 'fast-check';test('add: sums any list of non-negative integers', () => {  fc.assert(fc.property(fc.array(fc.nat(1000)), (nums) => {    const expected = nums.reduce((a, b) => a + b, 0);    assert.equal(add(nums.join(',')), expected);  }));});test('add: order does not matter', () => {  fc.assert(fc.property(fc.array(fc.nat(1000)), (nums) => {    assert.equal(add(nums.join(',')), add([...nums].reverse().join(',')));  }));});

fc.array(fc.nat(1000)) means "an array of whole numbers from 0 to 1000". Both passed.

Finding a real bug

Next, a price formatter. It has a bug that I put in on purpose, and the round trip property: formatting a number and parsing it back should give the same number.

// src/money.jsexport function formatCents(cents) {  return Math.floor(cents / 100) + '.' + (cents % 100);      // bug: no zero padding}export function parseMoney(text) {                            // "12.5" means 12 dollars 50 cents  const [whole, frac = '0'] = text.split('.');  return Number(whole) * 100 + Number(frac.padEnd(2, '0'));}

test('money: parse(format(x)) gives x back (round trip)', () => {  fc.assert(fc.property(fc.nat(100000), (cents) => {    assert.equal(parseMoney(formatCents(cents)), cents);  }));});

✖ money: parse(format(x)) gives x back (round trip) (3.678989ms)  Error: Property failed after 2 tests  { seed: -1130927682, path: "1:1:0", endOnFailure: true }  Counterexample: [1]  Shrunk 2 time(s)    10 !== 1

Read it from the top: it found a failing input after 2 tries, shrank it twice to the smallest it could, and the smallest is 1 cent. formatCents(1) gives "0.1", which parses as 10 cents. A person who wrote only the examples 105 and 1200 would never have seen this. The seed and path let you replay exactly this failure. I did that with { seed: -1130927682, path: '1:1:0' } as the second argument of fc.assert, and it failed again with Counterexample: [1].

The generator is random, so another run can start from a different number. With seed: 42 I got:

Property failed after 6 tests{ seed: 42, path: "5:4", endOnFailure: true }Counterexample: [27101]Shrunk 1 time(s)

Same bug, a larger counterexample (27101 is 271.1, parsed as 27110). So shrinking is a search that helps, not a promise of the absolute smallest case. Always copy the seed from a failure into your bug report or a regression example test.

The fix is one line:

return Math.floor(cents / 100) + '.' + String(cents % 100).padStart(2, '0');

✔ money: parse(format(x)) gives x back (round trip) (1.365215ms)ℹ pass 3

Good properties to look for

Tips: keep generators narrow enough to be valid (fc.nat(1000), not any number), keep properties fast, and keep a plain example test for each bug the tool found. Property tests complement example tests; they do not replace them.

4. Mutation testing with Stryker

Here is the question coverage cannot answer: if I introduce a bug, will a test fail? Mutation testing answers it directly:

1. The tool copies your code and makes one small change (a mutant): < becomes <=, + becomes -, an if condition becomes false, a string becomes another string.
2. It runs your tests against the mutant.
3. If a test fails, the mutant is killed (good). If all tests pass, it survived: your tests cannot see that change.

I ran StrykerJS on the String Calculator with this config (stryker.config.json):

{  "testRunner": "command",  "commandRunner": { "command": "node --test test/calc.test.js" },  "mutate": ["src/calc.js"],  "coverageAnalysis": "off",  "reporters": ["clear-text"],  "concurrency": 2,  "tempDirName": ".stryker-tmp"}

npx stryker run

Instrumented 1 source file(s) with 20 mutant(s)All tests  ✓ All tests (killed 17)[Survived] ConditionalExpressionsrc/calc.js:2:7-     if (s === '') return 0;+     if (false) return 0;[Survived] StringLiteralsrc/calc.js:2:13-     if (s === '') return 0;+     if (s === "Stryker was here!") return 0;[Survived] EqualityOperatorsrc/calc.js:4:40-     const negatives = nums.filter((n) => n < 0);+     const negatives = nums.filter((n) => n <= 0);All files |  85.00 |   85.00 |       17 |         0 |          3 |        0 |        0 |Done in 2 seconds.

My five tests killed 17 of 20 mutants: a mutation score of 85%. The three survivors teach two different lessons:

1. n <= 0 survived. No test has a zero in the input. A real bug could hide there: a version that rejected 0 as negative would pass my whole suite. The fix is a new test: add('0,5') should be 5. This is the normal use: a survivor points to a missing test.
2. if (false) return 0; and the string change survived. Why? Because the if (s === '') line is not needed: ''.split(...) gives [''], Number('') is 0, and the sum is 0 anyway. The mutant behaves exactly like the original. This is code the tests cannot tell apart from no code at all. Here the right action was to delete the line. (Such mutants are called equivalent mutants. They are a known annoyance of mutation testing, and not every survivor is a missing test.)

I added the zero test, removed the redundant line, and ran again:

✓ All tests (killed 16)All files | 100.00 |  100.00 |       16 |         0 |          0 |        0 |        0 |

16 of 16. Note the new total: removing the dead line also removed its mutants.

Cost and use

The whole run on this tiny file took about 4 seconds wall time here, with 20 mutants and a 0.2 s test command. The cost grows with (number of mutants) times (test time), so on a real project it can take many minutes. Teams usually run it on the changed files, or nightly, or only on core modules (pricing, permissions), and not on every commit. Stryker also has an incremental mode (the incremental option in its configuration) for that. I did not try it.

5. Coverage, and why 100% proves little

Code coverage tells you which lines ran. It does not tell you whether anything was checked. Here is a test with no assertions at all:

// weak/calc.weak.test.jstest('runs without crashing', () => {  add('1,2');  try { add('-1'); } catch {}});

I ran it with Node's built-in coverage:

✔ runs without crashing (1.440918ms)ℹ file               | line % | branch % | funcs % | uncovered linesℹ  calc.js           | 100.00 |   100.00 |  100.00 |

100% on lines, branches and functions, from a test that cannot fail on a wrong answer. Now the same weak test with Stryker:

All files |  37.50 |   37.50 |        6 |         0 |         10 |        0 |        0 |

Mutation score 37.5%: 6 mutants killed, 10 survived (the 16-mutant version of the file). Even the arithmetic a + b changing to a - b survived (ArithmeticOperator at src/calc.js:5:32). Same file, same "100% coverage", a very different quality. The good suite from section 2 scored 100%.

How to use coverage well:

• Low coverage is useful. A file at 10% has code nobody tested. Look at what is uncovered.
• High coverage is not proof. Do not use "80%" or "100%" as a goal for people, because people then write tests to satisfy the number (Goodhart's law: a measure that becomes a target stops being a good measure).
• Prefer a gate on new code ("changed lines must be covered") and a trend over a fixed global number.
• Look at branch coverage, not only lines: a ? b : c on one line can run only one side.
• Pair coverage with something that checks assertions: mutation testing on key modules, or code review of the tests (section 7).

6. A one-page team test strategy

A strategy is a short document that answers "what do we test, where, and what do we do when it breaks?" so that people do not decide again in every pull request. Copy this and fill it in. Keep it to one page, or nobody reads it.

# Test strategy: <team / service>      Owner: <name>      Reviewed: <date>## 1. What matters most (top 5 risks)1. <e.g. wrong price charged>      2. <e.g. user sees another user's data>3. <e.g. cannot log in>            4. ...## 2. Where we test it| Layer | Tool | What goes here | Target speed | Who writes it ||---|---|---|---|---|| Unit | node:test / Vitest | pure logic, edge cases | < 1 ms each | the author of the code || Component | Testing Library | rendering, user events | < 100 ms | frontend || Integration / API | supertest + real DB | routes, SQL, auth, webhooks | < 50 ms | backend || E2E | Playwright | max <N> journeys: login, checkout | < 10 s | shared || Contract | OpenAPI validation | API shape vs docs | fast | API owner |## 3. Rules- Bug fix = a failing test first (red), then the fix.- No sleeps. Wait for conditions. Control the clock.- Each test creates its own data. No test depends on another.- Fake only what we do not own (payments, email). Use a real DB for API tests.- New code: changed lines covered; critical modules reviewed with mutation testing <how often>.## 4. Pipeline- On every PR: lint, types, unit, component, integration (target < <N> min).- On main / nightly: full e2e on <browsers>, mutation run on <modules>.- Block merge on: any failing test. Do NOT block on: a global coverage number.## 5. Flaky tests- A flaky test is quarantined within <1 day>, gets a ticket with an owner, and is fixed or deleted within <2 weeks>.- Retries only in CI, with trace on first retry; a test that needed a retry is logged.## 6. What we do not test (on purpose)- <e.g. third-party widget internals, pixel-perfect layout, getters/setters>## 7. Review checklist for tests: see section 7 of the testing course

Keep the risks first (effort follows damage), the layer table with speeds (so nothing drifts into e2e), and a flaky policy with owners and dates (Lesson 4 shows how to fix them). "What we do not test" matters as much as what we do: it shows the choice was deliberate.

7. How to review tests

Reviewers often read the production code carefully and skim the tests. Tests deserve the same care. Questions I ask:

1. Would this test fail if the code were wrong? Mentally break the code (change < to <=, return a constant). If the test still passes, it is weak. This is mutation testing done in your head.
2. Did I see it fail? For bug fixes, the PR should show the test failing without the fix (the red step).
3. Does it check behaviour or implementation? A test that asserts which private function was called breaks on every refactor. A test that asserts the output or the saved data does not.
4. Is the name a sentence a person can read? rejects a request with no token beats test1.
5. Is it independent? Does it set up its own data? Could it run alone, in any order, in parallel?
6. Is it deterministic? Any sleep, real clock, random value without a seed, or call to the real network is a flaky test waiting to happen.
7. Are the assertions real? Look for tests with no expect/assert, try/catch that swallows errors, and assertions that cannot fail (expect(x).toBeDefined() on something always defined).
8. Is it at the right layer? A pure calculation tested through the browser should move down. A permission rule tested only with mocks should move up to an integration test.

Try it

1. Do the String Calculator kata in your own language. Add a custom delimiter (//;\n1;2). Write the test first, watch it fail, then write the code. Answer: the red message should mention NaN or a wrong sum, because ; is not split yet.
2. Write a property for reverse(reverse(list)) === list with fc.array(fc.integer()). Then write a buggy reverse that drops the last item when the list is longer than 5. Answer: fast-check should report a counterexample with at least 6 items (I did not run this exercise).
3. Take any small file you own. Change < to <= by hand and run your tests. Do they fail? Answer: if not, you found the same kind of gap Stryker found for n <= 0.

Interview questions

Key takeaways

• TDD is red, green, refactor in tiny steps. Always watch the test fail for the right reason.
• A bug fix starts with a failing test that reproduces the bug.
• Property-based testing checks a rule on many generated inputs and shrinks failures to a small counterexample. Save the seed.
• Mutation testing asks "would my tests notice a bug?" Survivors show missing tests, dead code, or harmless changes. Read each one.
• Coverage finds untested code. It cannot show that tests are good: one assertion-free test gave 100% coverage and a 37.5% mutation score.
• A one-page strategy (risks, layers, rules, pipeline, flaky policy, what we skip) saves arguments.
• When reviewing a test, ask: would it fail if the code were wrong?

What's next

This was the last lesson of the series. A good next step is to apply it to your own project: write the one-page strategy for it, pick one module and run mutation testing on it, and fix the three slowest or flakiest tests. Then go back to Lesson 0 and check whether your suite has the shape you want. For interview practice see Team Lead, Lesson 6.