Architecture, Data and Team Judgement Scenarios

This lesson has fourteen scenarios about architecture, data and working with people. Interviewers use them to see if you make good decisions when nobody tells you the answer.

A decision framework you can reuse

1. Clarify the goal and the limits. What problem are we solving, for whom, by when, and with which team?
2. List 2 to 3 options. Include "do nothing" and "the smallest change".
3. Compare the options on what matters: risk, cost, time to value, how easy it is to undo, the work to run it, and team skills.
4. Recommend one. Say what evidence would change your mind.
5. Plan the rollout. Use small steps, know how to roll back, and decide how to measure success.
6. Communicate to stakeholders (the people who care about the result) in terms of outcomes and risks.

A trade-off means you gain one thing and lose another. A rollback means going back to the last good version.

The scenarios

1. "Deploys of our monolith are slow and risky. Should we move to microservices?"

A monolith is one application that you build and deploy as a whole. Microservices are many small services, each deployed on its own and talking over the network.

Find the real problem first. Slow and risky deploys can come from missing tests, a slow pipeline, tight coupling or no feature flags (switches that turn a feature on or off without a new deploy). Microservices do not fix any of these by themselves. Microservices also bring network failures, data spread over many places, contracts with versions, the need to watch many services, and much more operations work.

Options:

• (a) Improve the monolith: faster CI, tests, flags and canary deploys (send a small part of traffic to the new version first).
• (b) Make a modular monolith, which is one app with clear parts and rules that keep the boundaries.
• (c) Take out one or two services where there is a real need, such as a different scaling profile, a separate team, or a risky part that needs isolation.
• (d) Go to full microservices.

Recommendation: start with (a) and (b). Take out a service only with a clear reason, behind a stable interface, using the strangler approach (replace the old system piece by piece). Measure deploy frequency (how often you ship), lead time (time from commit to production) and change-failure rate (the share of deploys that cause a problem) before and after.

2. "The legacy system is a mess. Rewrite it or refactor?"

Choose small steps by default. A rewrite gives no value until the end. It is hard to estimate. It often brings back old bugs, and the old system keeps changing while you build. Fred Brooks called this the "second-system effect": the team tries to put in every idea it could not use before.

Strangler fig pattern (named after a vine that slowly grows around a tree and replaces it):

1. Put a facade or router (one front door that sends each request to the old or the new system) in front of the old system.
2. Build new features, or rebuild one slice, in the new system.
3. Move traffic over slowly, by feature, route, tenant or percentage.
4. Keep both systems running.
5. Retire parts of the old system when nobody uses them any more.

Before you start, add characterisation tests (tests that record what the old system does today) around the legacy behaviour.

When a rewrite makes sense: the platform has reached end-of-life and has no migration path, the domain is small and well understood, or keeping the old system costs more than a full rebuild. Even then, ship in vertical slices (small pieces that work from top to bottom).

3. "Pages are slow, so the team wants to add Redis caching everywhere."

Ask first: why is it slow? Measure it (Lesson 00). Caching means keeping a copy of a result so you do not compute it again. It can hide a slow query. It also adds invalidation (knowing when to throw away old copies) and consistency problems (different parts of the system showing different data).

If caching is a good idea, decide these things:

• what to cache: data that is read a lot, costly to make, and fine if slightly old
• where: CDN (servers around the world that keep copies), Redis (a fast in-memory store) or in-process memory
• the key design
• the TTL (time to live: how long a copy may be kept) and invalidation plan: clear on write, or use versioned keys (put a version number in the key, so a new version means a new key)
• stampede protection (stop many requests rebuilding the same value at once)
• what must never be cached: personal data or data that must always be exact

Add a metric for the hit ratio (how often the cache has the answer). Add a way to skip the cache when you debug. (Lesson 02)

4. "We need to rename a column (or change its type) on a 200 GB table with zero downtime."

Use expand and contract (also called parallel change). It means: first add the new thing, then move code over, then remove the old thing. The rule: never make a change that old code and new code cannot both live with. A plain RENAME COLUMN is fast in the database, but it breaks every running copy of the old code at once. Changing a type, such as int to bigint, usually rewrites the whole table and holds a strong lock.

1. Expand: add the new column (nullable; in PostgreSQL 11 and later a constant default is also instant, but a changing default such as random() rewrites the whole table). Do not remove the old one.
2. Dual-write: deploy code that writes to both columns (or use a trigger, which is code that the database runs by itself on each write).
3. Backfill (fill in the new column for the existing rows) in small batches with throttling (pausing between batches to keep the load low). The job must be idempotent and able to resume.
4. Verify that the two columns match (a reconciliation query, row counts, checksums).
5. Switch reads to the new column behind a feature flag. Watch the metrics. Keep the option to switch back.
6. Contract: stop writing the old column. Then drop it in a later release, after a safe waiting time.

Safety rules:

• Each step is a separate deploy, and each can be undone.
• Use online DDL (DDL is the SQL that changes table structure; "online" means it does not block normal queries for long). For example, CREATE INDEX CONCURRENTLY (it cannot run inside a transaction, and if it fails it leaves an invalid index that you must drop), then NOT VALID constraints followed by VALIDATE CONSTRAINT.
• Set lock timeouts (the migration gives up if it cannot get its lock quickly, instead of making other queries wait).
• Practise on a copy that is as big as production.
• Run it when traffic is low.

The same pattern works for table splits, type changes and moving data between services. (Lesson 11, scenario 11)

5. "Backfill 100 million rows without hurting production."

Approach: run it as a throttled background job that can resume.

• Work in batches by primary-key ranges (for example id 1 to 10,000, then 10,001 to 20,000), not with OFFSET.
• Commit each batch. Do not use one giant transaction: it holds locks, blocks cleanup (vacuum) and can fill the disk with old row versions.
• Pause or slow down depending on database load and replica lag (how far a replica is behind the primary).
• Make each batch idempotent.
• Save the progress (a cursor), so a restart carries on from the same place.
• Run it when traffic is low, if you can.
• Write to the primary database, because a replica is read-only.
• Monitor lock waits, replication lag and the error rate.
• Add a kill switch (a way to stop it at once).

Before you cut over, check the results with sampling and counts.

6. "Reads are overwhelming the primary database. How do you scale?"

In this order:

1. Fix the queries and indexes. Look for N+1 queries and a pool that is too small or too big. Add caching for hot reads (data that is read very often).
2. Use read replicas (copies of the database that only serve reads) for read-heavy endpoints.
3. Partition big tables (split one table into smaller parts inside the same database) or archive old data (move it to cheaper storage).
4. Shard (split the data across several separate databases, each holding a part) only when one primary cannot handle the writes or the size. The primary is the main database that accepts writes.

Replicas bring replication lag. The copy is a little behind the primary. So send only reads that can accept old data to replicas. Send read-your-own-writes to the primary (or use a session or LSN-based consistency token; an LSN, log sequence number, is a position in PostgreSQL's write log, so you can wait until the replica has reached it). Monitor lag and set alerts. (Backend, Lesson 07)

7. "We are multi-tenant. How do we make sure one customer never sees another's data?"

Multi-tenant means many customers (tenants) share one system.

Layers of defence:

1. A tenant ID on every row and in every query. Enforce it in one central place, not by habit.
2. Row-level security in the database (PostgreSQL RLS: rules in the database that decide which rows each user may see) as a safety net. Remember: the table owner and superusers skip RLS unless you use FORCE ROW LEVEL SECURITY, so the app must connect as a normal role. With a connection pool, set the tenant for each transaction (SET LOCAL), not for the whole connection.
3. Take the tenant from the verified session (the login data the server has already checked), never from the request body.
4. Separate caches and keys for each tenant (tenant:{id}:…). Use separate encryption keys for sensitive data.
5. Tests that try cross-tenant access on every endpoint.
6. Audit logs and alerts for unusual activity.
7. Think about a schema per tenant or a database per tenant. This gives strong isolation, or control over a "noisy neighbour" (one tenant that uses too many resources). It costs more to run.

Also: set rate limits and quotas per tenant, so one customer cannot starve the others.

8. "The team has a lot of tech debt. How do you prioritise it and convince the business?"

Tech debt means shortcuts in the code that make future work slower, like a loan that you pay back with interest.

Make it visible and link it to outcomes. List the debt items with their impact (number of incidents, slow delivery, onboarding time, security risk) and the cost to fix. Rank them by interest rate. Debt in code that you change every week costs more than debt in a quiet corner. Explain the case in business terms. For example: "This module causes 40% of our bugs and adds two days to every change. Fixing it saves about X per quarter."

How to schedule it: keep a fixed share of capacity (say 15 to 20%). Do boy-scout fixes (leave the code a bit cleaner than you found it) next to feature work. Add a few refactoring projects with a clear scope and clear exit criteria. Avoid "big-bang" cleanups that show nothing for a long time.

9. "A stakeholder demands a date that you believe is unrealistic."

Do this:

1. Understand why the date matters.
2. Break the work into parts. Estimate with ranges and assumptions.
3. Find the critical path (the chain of tasks that sets the end date) and the risks.
4. Give options, not only "no": reduce the scope (an MVP, a minimal first version), move the date, add people (and count the cost of onboarding them), or accept the risk with a plan to reduce it.
5. Agree what is cut first if the work slips. Estimate again as you learn.
6. Keep a record of the decision.

Avoid: agreeing quietly and burning out the team (people become exhausted from too much work), or hiding extra padding in the estimate without saying why.

10. "In code review, a teammate and you disagree on an approach."

Aim for the best result, not for winning.

• Make sure you understand their reasons.
• Separate taste from correctness or risk.
• Use data (benchmarks, a small prototype) or the team's standards and ADRs (written records of past decisions).
• Offer a compromise or an experiment.
• If you are still stuck, set a time limit and ask a tech lead. Or decide with an agreed principle, such as how easy it is to undo, or consistency with existing code.
• Once it is decided, disagree and commit: support the decision even if you did not choose it. Write down the decision.

Keep your comments about the code, and keep them kind and specific.

11. "Should we build our own auth/search/billing or buy a service?"

Compare these points:

• strategic value: is this core to our product?
• total cost: build, run, maintain, security and compliance
• time to market
• fit and flexibility
• lock-in and the cost of leaving
• reliability and support
• data residency and compliance

Default: buy a service or use open source for common capabilities (auth, email, payments, search). Build what makes you different from others. Reduce lock-in with an abstraction layer and a clear way to export your data. Before you commit, run a short proof of concept with real requirements and edge cases.

12. "We have almost no monitoring. Where do you start?"

In order of value:

1. Uptime and error-rate checks with alerts on symptoms that users can see: synthetic checks (fake users that test your site), the 5xx rate (server errors) and latency percentiles (for example p95: 95% of requests are faster than this value).
2. Structured logs with request IDs and error tracking.
3. The RED metrics (Rate, Errors, Duration) for each service, and key business numbers (orders, signups).
4. Resource metrics: CPU, memory, DB connections, queue depth (how many jobs wait) and event-loop lag (how late Node's main loop runs its tasks).
5. Distributed tracing (following one request through many services).
6. SLOs (service level objectives, your targets for reliability, such as "99.9% of requests succeed") and error budgets (how much failure the SLO allows, which tells you when to slow down changes).
7. Runbooks (step-by-step guides) linked from alerts, and an on-call rotation.

Start small. Five good alerts are better than fifty noisy ones. (Backend, Lesson 05)

13. "Our CI takes 40 minutes and developers stopped waiting for it."

Measure where the time goes, for each job and step. Then fix the biggest parts:

• Cache dependencies and build outputs.
• Run jobs in parallel and shard tests (split the test suite into groups that run on several machines at once).
• Run only the affected tests and builds. A monorepo is one repository that holds many projects. There, use tools such as Nx or Turborepo with remote caching (build results are saved on a server and reused).
• Move slow end-to-end tests (tests that drive the whole app) to a post-merge (after the code is merged) or nightly stage. Keep a fast pre-merge smoke set (a small set of key checks), and be ready to revert when a post-merge run fails.
• Run the cheapest checks first (lint, types, unit tests), so most failures show up in minutes.
• Use bigger or faster runners.
• Fix flaky tests (tests that pass and fail at random), or quarantine them (run them apart so they do not block merges).
• Keep Docker layers cacheable (put the steps that change rarely, such as installing dependencies, before the steps that change often).

Set a target (for example under 10 minutes) and track it.

14. "A bug that you introduced reached production and cost the company money."

Own it, fix it, learn from it.

• Contain it first (roll back or turn the flag off).
• Communicate clearly and quickly, with no excuses.
• Fix the data if it was affected.
• Join a blameless postmortem (a review that looks for causes, not for someone to blame). Find why the system allowed it, such as a missing test, an unsafe default or no canary. Then add a guard.

In an interview, show responsibility and growth. Say what happened, what you did in the first hour, and what changed afterwards. Do not blame others and do not play down the impact. (Lesson 10)

Interview questions

Key takeaways

• Use a framework: clarify, options, trade-offs, recommendation, rollout and rollback, communication.
• Small steps beat big-bang changes: strangler fig, expand and contract, throttled backfills, and a modular monolith first.
• Measure before you change (caching, scaling, CI) and link decisions to outcomes.
• Interviews test soft skills too: offer options when you negotiate scope, disagree and commit, and own your mistakes.

What's next

That finishes the Scenario Debugging series. Go back to Lesson 00 for the method. Then keep practising the banks in Lessons 11, 12 and 13, until your answers always come out in the same three parts.