Scaling Teams, Ownership and On-Call

A team of four shares everything in their heads. At fifteen people nobody knows who owns the billing job that failed at 3 a.m. At forty, two teams change the same module and each blames the other for the outage. This lesson covers how to split teams, how to write down ownership, how on-call works, how to avoid heroes, and how to manage dependencies between teams.

Read Laws and biases, Lesson 7 for Conway's Law and Dunbar's number, and Incidents, team health and growing as a lead, Lesson 4 for incident roles and delivery measures. Technical monitoring is in Full-Stack Lesson 26 and AI Engineering Lesson 4. Runbooks and incident reviews tie in with Lesson 17.

Everything here is common practice and opinion. Companies and teams differ a lot. Use the ideas, not the exact words.

Words used in this lesson

• Ownership means a named team (and, inside it, a named person) answers for a service or area: its quality, its incidents and its future.
• Service catalogue is a list of every service or system with its owner, purpose, links to its docs and its dashboards.
• On-call means being the person who is reachable to respond when something breaks outside normal hours or between tasks.
• Rota (rotation) is the schedule of who is on call when.
• Runbook is a short document with the steps to handle a known problem.
• Toil is repetitive manual work that grows with the service and could be automated.
• Bus factor is how many people can leave before something has no one who understands it.
• Hero culture is when the same few people save every crisis and the team depends on that.
• Inverse Conway manoeuvre means designing the teams to match the software you want, instead of letting the existing team shape decide your architecture.

Splitting teams, and why Conway matters

From Lesson 7: Conway's Law says a system's design tends to copy the communication structure of the organisation that builds it. The list of reading I used makes the point from the other side. Its entry on Conway's Law in Team Topologies is described this way: the "Reverse or Inverse Conway Maneuver" implies that we should design our teams (not the software yet) to get the software we want. So team boundaries are an architecture decision.

The same list has many articles on team structure (most are title only, so I do not describe them): Martin Fowler's Team organization articles, Team Topologies (it describes Fowler's summary of the book as "great"), Organizing software teams, Independence, autonomy, too many small teams, Small teams from an employee handbook, Infrastructure Platform Engineering Organizational structures and models (called "an incredible article" about structuring an infrastructure team), and Spotify's Failed #SquadGoals. The last title is a good warning in itself: copying another company's named model does not copy its context.

Principles that most of these sources and my own experience agree on (opinion):

1. Keep teams small enough to talk. A rule of thumb you may know is "two pizzas" (from the laws list in Lesson 7); Dunbar's number shows that people can only keep a limited number of working relationships. Fewer than ten is a common target.
2. Split along the work, not along job titles. A team that can deliver a change from idea to production without waiting for another team moves faster than "frontend team", "backend team" and "database team".
3. Give each team a clear mission and a clear boundary. If you cannot say in one sentence what the team is for, the boundary is wrong.
4. Reduce hand-offs. Each hand-off is where requirements get lost and queues form.
5. Split when the pain is real: when one team is always blocked by itself, when two parts have different release speeds, or when ownership is unclear and bugs fall between chairs. Do not split because the headcount reached a number.
6. Do it in steps and explain why. Reorganising causes anxiety. Say what problem you are solving, how people will be involved, and what will not change.

Ownership you can see

If nobody knows who owns a thing, nobody maintains it. Write ownership down:

A service catalogue can be as simple as a table in the repository:

| Service | Purpose | Owner team | On-call | Docs | Dashboard | Tier ||---|---|---|---|---|---|---|| checkout-api | Takes orders | Payments | payments-oncall | /docs/checkout | link | 1 (revenue) || email-worker | Sends receipts | Platform | platform-oncall | /docs/email | link | 2 || internal-admin | Back-office UI | Support tools | business hours | /docs/admin | link | 3 |

Rules that keep it alive:

1. One owner per service, not "everyone". A group can own it, but a person is the contact.
2. Put the file next to the code and check it in reviews. A stale catalogue is worse than none.
3. Tier the services so you know which ones justify a page at night and which can wait until morning.
4. CODEOWNERS files (see Git, Lesson 5) can send reviews to the right team automatically.
5. Check ownership when people leave. Orphaned services are how incidents turn into long ones.

The reading list also links to articles on production ownership. One in the on-call section, Production Oriented Development, is described as arguing that engineers should own and operate the code they write in production. That idea is a common reason to make developers part of the on-call rota rather than hand the pager to a separate team. (It is one view and it has costs, discussed below.)

On-call that works

On-call is how a team honours its promise to users. Done badly, it burns people out. The list I used has several useful pointers (descriptions are the list's):

• Alert on symptoms rather than causes. From the article My Philosophy On Alerting: a page should mean "users are hurt or about to be", not "a disk is at 80%".
• Remove noisy alerts. The Paradox of Alerts is described as arguing that deleting most paging alerts can make systems better. Every false page teaches people to ignore the next one.
• Tie monitoring to business metrics (from a PagerDuty article on business metrics and outage response).
• Default to rolling back rather than fixing forward during an incident (from Testing in Production: the hard parts).
• Measure incidents with clear terms. Incident Metrics in SRE is described as defining time to detect, time to mitigate, time to recover and time between failures.
• Use an incident commander. The incident response reading includes Incident Response at Heroku, described as covering the Incident Commander role, inspired by disaster response.
• Learn from the on-call chapter of the Google SRE Workbook (it is on the list; I could not open it, so I do not describe it).

A humane on-call setup (my own checklist; the numbers are illustrations):

Avoiding hero culture and the bus factor

• Notice the hero. If the same person fixes every crisis, ask what knowledge sits only in their head. Thank them and then spread the knowledge.
• Pair on incidents. Put a second person with the on-call, especially a newer one. Rotate who leads.
• Write it down. Runbooks, architecture notes and decision records (Lesson 5).
• Count the bus factor for each service in the catalogue. If it is one, name a second person and give them a real task.
• Don't reward heroics, reward prevention. Praise the person who removed the cause of the 3 a.m. alert more than the person who woke up and fixed it again.

Toil

Toil is the quiet cost of growth: manual restarts, hand-edited config, copying data by hand, answering the same question. Habits:

1. Name it and count it. Ask the team to note in a shared list every time they do a repetitive manual task.
2. Reserve time for it. A fixed share of each cycle for removing toil is easier to defend than a one-off project.
3. Automate the top item, then re-count.
4. Do not automate what you do not understand yet. Write the steps as a runbook first; the runbook is the automation's specification.

Dependencies between teams

When teams are separate, the connections between them are the risk. Practical habits:

• Treat other teams like customers with a contract: a documented interface, a version, a way to ask for changes and a way to report problems.
• Make requests visible: a shared queue or board, not a private message.
• Say what you need and by when, with the reason.
• Plan cross-team work early: list dependencies at the start of a quarter, with a named contact on each side.
• Don't solve a dependency by going around it (copying the other team's data, editing their code without agreement). Solve it with an agreed interface, or with an agreed change to the boundary.

The reading list has an article on Organizational boundary problems: too many cooks or not enough kitchens?, described as "a lot of useful resources to design organizations". I did not open it. Its title makes a useful pair of questions for your own boundaries: is a task shared by too many hands, or does a team lack the resources to finish it alone?

Scaling the lead role

As the organisation grows, the lead's job moves:

• From deciding to setting context. You cannot be in every decision; give teams the goals and limits so they decide well.
• From knowing everything to knowing who knows. Keep the catalogue and the contacts up to date.
• From doing to enabling. Remove blockers, fix the process, and grow future leads (Lesson 13).
• From talking to everyone to a few reliable channels: a weekly written update, a regular all-hands, and 1:1s with your direct reports (Lesson 8).

The list has two scaling articles in its "Scaling an organization" section, both title only: Scaling Without Imploding and Stripe's Atlas: Guide to scaling engineering organizations. They are good next reads; I cannot say what they conclude.

Interview questions

Key takeaways

• Team boundaries are an architecture decision, because software tends to copy communication structure. Design teams to match the software you want.
• Keep teams small, split along the work, give each a clear mission and cut hand-offs. Copy the reasoning of other companies, not their model.
• Write ownership down in a service catalogue with one owner per service, a tier and an on-call contact.
• Page only for user-visible symptoms. Review pages monthly, delete the noisy ones, and give each remaining alert a runbook.
• Spread knowledge to avoid heroes and a bus factor of one. Reward prevention.
• Count and remove toil with reserved time.
• Treat other teams as customers with a contract: documented interfaces and visible requests.
• This is common practice and opinion. Adapt it to your team.

What's next

Next: Culture, psychological safety and retrospectives, Lesson 17. Practise: build the service catalogue table for the services your team touches. Count how many have no clear owner or a bus factor of one.