Technical Debt, Quality and the Rewrite Trap

"Why is this code so messy?" "Can we rewrite it?" "Why do we need tests for this?" "Why did production go down?" These questions land on the lead. This lesson gives simple ways to think about debt, refactoring versus rewriting, test strategy, what to watch in production, and what real outages teach.

Read Lesson 0 first for the answer shape. For the quick fix versus proper fix questions see Lesson 3. For testing tools see Full-Stack Lesson 21, and for monitoring see Full-Stack Lesson 26.

Everything here is common practice and opinion. Teams differ. Use the ideas, not the exact words, and always use your own true examples.

Words used in this lesson

• Technical debt is a shortcut that saves time now and costs time later, like a loan with interest.
• Refactoring is changing the inside of code to make it easier to understand, without changing what it does.
• A rewrite is throwing away working code and building it again from scratch.
• A codemod is a script that changes code automatically across many files (for example to move to a new API).
• A test pyramid is the idea of having many small fast tests, fewer medium ones, and few slow end-to-end tests.
• A flaky test sometimes passes and sometimes fails without a code change.
• An SLO (service level objective) is a target for how well a service should work, such as "99.9% of requests succeed". An SLI is the measure it uses.
• A rollback is going back to the previous working version.

Technical debt: a loan, not a sin

The debt picture has some useful points from the reading list and from search summaries of Martin Fowler's work:

• The word "debt" was a metaphor from Ward Cunningham about building a product step by step as you learn. The list notes that this kind of debt is often invisible to static analysis tools: a tool cannot see "this model no longer matches how the business works".
• Debt can be a good deal. Fowler's quadrant (as search summaries describe it) sorts debt by two questions: was it deliberate or accidental, and was it prudent or reckless? A deliberate, prudent choice ("we ship the simple version now and fix it in two weeks") is a fair trade, if you really pay it back. Reckless debt ("we have no time for design") has heavy ongoing costs. Prudent debt in code nobody touches may not be worth paying at all, because the interest is small.
• You can keep shipping while you pay it back. One article in the list ("Fixing Technical Debt with an Engineering Allocation Framework") says teams can do both, if they explain the business value of the clean-up.

A simple way to work with debt:

1. Write it down when you take it: a ticket, a TODO with a link, or a line in the decision note. Debt that nobody wrote down is just a surprise.
2. Say what it costs. "Each new report takes three days instead of one because of this tangle." Product people can use that.
3. Pay in small steps, near the code you are already changing. The Scout Rule from hacker-laws says: leave the code cleaner than you found it.
4. Pay the expensive debt first. Pay debt in code that changes often, or that causes incidents, not debt in code that never changes.

Refactor or rewrite?

The reading list has one dedicated item on this ("Refactor vs. Rewrite", title only, so I do not know its argument). The nearby advice is clear though:

• Joel Spolsky, "Things You Should Never Do, Part I" (listed in the technical strategy section) argues against rewriting a working codebase from scratch.
• Gall's Law (hacker-laws): complex working systems grow from simpler working systems, and a complex system designed from scratch tends not to work. So grow from what works.
• Chesterton's Fence (hacker-laws): do not remove or change something until you understand why it exists. Old ugly code often holds a bug fix you have forgotten.
• The Rule of Three (listed in the refactoring section): build the reusable version after you have seen the same need in three places, not after one.
• Codemods (listed, from Martin Fowler's site): for a big API change, automate it with a script instead of editing hundreds of files by hand.

My rule of thumb (opinion): refactor in small steps with tests, keep the system running the whole time, and consider a rewrite only for a small, well-understood part with a clear reason you can measure.

Quality is more than clean code

An article in the list ("Things end users care about but programmers don't") says quality includes the user experience, formatting, compatibility and how the thing is run, not only the code. For a lead that means asking: does it work for users, can we see when it breaks, and can we undo it?

Two small ideas from the same section:

• Use your type system. Give things their own types (a UserId is not an OrderId) instead of passing bare numbers and strings around.
• Test with a plan. The reading list points to several well-known views: The Practical Test Pyramid (agree on test types and scope, and treat test code as seriously as production code), Write tests. Not too many. Mostly integration., Google's advice to say no to more end-to-end tests because they are slow and brittle, and Eradicating Non-Determinism in Tests (remove the causes of flakiness). These views disagree in places, which is fine. Pick a shape that fits your system, and say why.

What to watch in production

You cannot fix what you cannot see. The list points to the Google SRE book's monitoring advice: watch latency, traffic, errors and saturation (how full the system is). Other short ideas from the list:

• Meaningful availability. A good availability number reflects what users feel, changes in proportion to the problem, and points to causes.
• Own what you ship. One article argues that engineers should own and operate the code they write in production.
• Default to rolling back during an incident, instead of fixing forward under pressure.
• Tie alerts to business impact so people are woken only for things that matter.

Learn from real outages

The community list danluu/post-mortems collects public postmortems and groups them (Config Errors is the first and largest group, then Hardware/Power, Conflicts, Time, Database and others). Five well-known ones, as the list summarises them:

Reading three of these a month is a cheap way to practise for the "tell me about a production incident" question.

Interview questions

Key takeaways

• Technical debt is a loan with interest. Take it on purpose, write it down, show the cost in business terms, and pay it in small steps.
• Pay first the debt in code that changes often or causes incidents.
• Prefer refactoring in small steps to a full rewrite. Remember Gall's Law and Chesterton's Fence.
• Quality includes what users feel, how the thing is run, and whether you can roll back.
• Keep a test plan: many fast tests, some integration tests, few end-to-end tests. Fix flaky tests fast.
• Watch latency, traffic, errors and saturation. Prefer rolling back to fixing forward under pressure.
• Read public postmortems. Many are about configuration changes, so treat configuration like code.
• This is common practice and opinion. Adapt it to your team.

What's next

Next: Laws and biases every lead should know, Lesson 7. Practise: list the three biggest pieces of debt in your project, write the cost of each in days per month, and decide which one you would pay first and why.