Part 11 · Design It Yourself: Case Studies

What I Learned From Big Tech Engineering Blogs

Interview designs are clean; real systems are messy, migrated and patched for years. How to read engineering blogs and postmortems so you actually learn from them, the ten lessons that keep repeating across big-tech stories, and a reading list to keep going.

Lesson 6510 min readall#case-study#learning

The Problem

Interview-style designs are clean: boxes, arrows and a neat final answer. Real systems are messy:

they were built by teams under deadlines, with the knowledge they had at the time,
they grew in ways nobody predicted,
they broke in surprising ways,
and they were migrated, rewritten, un-rewritten and patched for years.

The best way to learn how systems really change over time is to read the stories engineers publish themselves. These are engineering blogs, conference talks, research papers and public postmortems. A postmortem is a written report about an outage. It says what happened, why it happened, and how to stop it happening again. Throughout this series, I have pointed to many of these stories (Discord, Stripe, Netflix, Cloudflare, Instagram, Notion, GitHub, Google, Amazon). In this post, I will share how to read them, the lessons that keep repeating, and a reading list to keep learning after the case studies.

The Core Idea

Reading engineering blogs is like reading the travel diaries of people who have already climbed the mountain you are about to climb. The diaries tell you:

which paths looked easy but led to cliffs,
where the weather turned bad,
which equipment actually mattered,
and, most importantly, why they made each choice, given their conditions.

You won't climb exactly the same route (your team, traffic and budget are different), but you'll recognise the dangers earlier and make better decisions.

How It Works

How to read an engineering post (and actually learn from it)

Do not just look for "they used Kafka". For each post, write down five things. Kafka is a system that stores a stream of events in order, so many programs can read them.

1
Problem
what was broken or about to break — with numbers
2
Constraints
scale, team size, budget, deadlines, existing tech, regulations
3
Options
the alternatives considered, and why they were rejected
4
Decision
what they chose, and the trade-offs they accepted
5
Result
what improved, what went wrong, what they'd do differently
6
Then ask
would this apply at my scale, with my team? which building block is it really about?

Five notes to take on every engineering post — and the question to ask afterwards. · flow

Then ask yourself:

Would this apply to my system? At my scale, with my team?
Which building block from this series is this really about? For example: caching (keeping a quick copy of data), sharding (splitting data across machines), idempotency (an action is safe to repeat), or load shedding (refusing some requests so the system survives).
What is the "boring" lesson behind the exciting headline?

Be a little sceptical (do not believe everything):

Blog posts are written after success, and often leave out the messy parts.
Solutions are tuned to that company's scale. What Netflix needs may be overkill for you.
Posts get old. Check the date, and check whether the company later changed its approach. Many did.

The lessons that keep repeating

After reading many engineering blogs and postmortems, the same themes appear again and again.

1. Start simple, and scale when the numbers say so. Instagram, Stack Overflow, Shopify, Notion and many others grew very large on relational databases (databases with tables and SQL), caching and replicas (extra copies of the database) before they tried anything unusual. Notion and Figma only sharded Postgres (split it across many machines) when their single database was clearly reaching its limits (post 25). Stack Overflow is famous for serving a very large audience with a small number of powerful servers (post 7).

Lesson: the simplest thing that meets the requirements usually wins, as long as you measure and plan the next step.

2. Migrations are the real work. Discord went from MongoDB to Cassandra to ScyllaDB. Facebook Messenger moved from HBase to MySQL/MyRocks. Uber moved some systems from Postgres to MySQL. Dropbox moved file storage off Amazon S3 to its own system. Every one of these posts spends more time on how they migrated safely than on the new technology (post 29). The safe steps include: dual writes (write to the old and new system at the same time), backfills (copy the old data over), shadow reads (read from the new system quietly and compare the answers), verification, and a gradual cut-over (move users over a little at a time).

Lesson: design for change, and learn the migration playbook, because you will need it.

3. Architecture swings back and forth. A monolith is one big application. Microservices are many small applications that talk to each other. Amazon and Netflix moved from monoliths to services. Segment moved back from 100+ microservices to one service. Amazon Prime Video moved one monitoring tool from many distributed serverless components to a single application. It cut the cost of that tool by about 90%. Uber grouped thousands of microservices into domains (post 53).

Lesson: there's no permanently "right" architecture, only the right one for your current team, scale and workload.

4. Caching is everywhere, and so are its bugs. Facebook's memcache paper, Twitter's Redis timelines, Instagram's Redis tuning and Etsy's cache smearing (posts 15–16, 64) all show that caching makes huge systems possible. Cache smearing means spreading out when cached items expire, so they do not all expire at once. Facebook's 2010 outage shows that caches can cause outages too. A bad setting made many clients ask the database for data at the same time. Each failed attempt made the overload worse, which is a feedback loop.

Lesson: cache a lot, but design for stampedes, invalidation and cache failure. A stampede is when many requests all miss the cache together and hit the database. Invalidation means removing or updating cached data when the real data changes.

5. Most outages start with a change. Across public postmortems (Cloudflare's regex incident, Knight Capital's deployment, CrowdStrike's content update, the AWS S3 typo, GitHub's failover in 2018, GitLab's database deletion), the trigger is usually a deploy, configuration change, command or automated action (posts 45, 48 and 58).

Lesson: small, gradual, observable, reversible changes prevent more outages than any other single practice. A canary release sends a new version to a few users first. A feature flag is a switch that turns a feature on or off without a new deploy. A staged rollout grows the number of users step by step.

6. Failure handling is where the design really lives. Netflix built Chaos Monkey (a tool that randomly turns off servers to test the system) and Hystrix (a library that stops calls to a failing service). AWS writes about static stability (keep working even if a control service fails), shuffle sharding (give each customer a random group of servers, so one bad customer hurts fewer others) and load shedding. Google's SRE books focus on overload and cascading failures, where one failure causes the next (posts 40–44).

Lesson: assume every dependency will be slow or down, and decide in advance what your system does then. Use timeouts (stop waiting after a set time), retries with jitter (wait a random extra time before trying again), circuit breakers (stop calling a service that keeps failing), bulkheads (keep resources separate so one failure cannot use them all), fallbacks (a simpler backup answer) and load shedding.

7. Idempotency and "exactly-once" are business problems, not just technical ones. An action is idempotent if doing it twice has the same result as doing it once. Exactly-once means every message is processed one time, no more and no less. Stripe's writing on idempotency keys, Confluent's writing on Kafka's exactly-once semantics, and many payment-system postmortems show that duplicate or lost messages cost real money (posts 34 and 37).

Lesson: make retries safe everywhere, and never trust a label that says "exactly-once" without asking "what happens if this runs twice?".

8. Observability pays for itself. Observability means being able to see what is happening inside your system from its logs, metrics and traces. Google's Dapper paper (about tracing one request across many services), the birth of Prometheus (a metrics tool) at SoundCloud, Honeycomb's writing on high-cardinality events (data with very many different values), and SRE practices around SLOs and error budgets all point the same way (posts 46–48). An SLO is a reliability goal, such as "99.9% of requests succeed". An error budget is how much failure the SLO still allows.

Lesson: you can't fix what you can't see. Instrument early, alert on user symptoms, and use SLOs to balance speed and reliability.

9. Security failures are usually basic. Equifax (software that was not patched), Capital One (SSRF, where an attacker tricks a server into calling internal addresses), Uber and Toyota (leaked secrets), MOVEit (SQL injection, where input is run as database commands) and Log4Shell and xz (attacks through dependencies, also called the supply chain) were rarely clever tricks (posts 49–52).

Lesson: patch, scan, least privilege, no secrets in code, and check authorisation on every object. Least privilege means giving each user or service only the access it needs. The basics stop most attacks.

10. People and culture matter as much as technology. Amazon's two-pizza teams (teams small enough to feed with two pizzas), Etsy's blameless postmortems (reports that look for system causes, not people to blame), Google's design docs and SRE error budgets, and Conway's Law (post 53) all show that how teams are organised and how they learn shapes the systems they build. Conway's Law says a system tends to copy the communication structure of the team that builds it.

Lesson: blameless learning, clear ownership and written decisions are part of system design.

1 · Start simple

scale when the numbers say so

2 · Migrations

are the real work

3 · Architecture

swings back and forth

4 · Caching

everywhere, bugs included

5 · Changes

cause most outages

6 · Failure handling

is where the design lives

7 · Idempotency

is a business problem

8 · Observability

pays for itself

9 · Security failures

are usually basic

10 · People & culture

shape the system

The ten lessons, compressed. · numbers

A reading list to continue

Here are some engineering blogs and resources worth following. Search the web for them by name.

Engineering blogs:

Netflix Tech Blog: resilience, chaos engineering, data pipelines, streaming at scale.
AWS Builders' Library: how Amazon builds and operates services (timeouts, load shedding, deployments, caching).
Cloudflare Blog: networking, edge computing, DDoS defence, and detailed public postmortems.
Discord Blog (engineering posts): real-time systems, Elixir, Rust, message storage at scale.
Stripe Engineering Blog: API design, idempotency, rate limiting, reliability.
Uber Engineering Blog: microservices at scale, Kafka, data platforms, geospatial systems.
Meta Engineering Blog: huge-scale infrastructure, caching, storage and AI systems.
Slack Engineering, GitHub Engineering, Shopify Engineering, Figma Engineering, Notion's engineering posts: product-scale systems and migrations.
LinkedIn Engineering, Pinterest Engineering, Airbnb Engineering, Dropbox Tech: feeds, search, data and infrastructure.

Curated collections (on GitHub):

awesome-scalability, an organised index of engineering posts by topic (scalability, availability, stability, performance),
The System Design Primer, including its lists of real-world architectures and company engineering blogs,
engineering-blogs, a long list of company engineering blogs to follow,
post-mortems, a collection of public incident reports,
papers-we-love, research papers grouped by topic.

Classic research papers (the ideas behind many systems in this series):

The Google File System (2003) and MapReduce (2004): large-scale storage and processing.
Bigtable (2006) and Spanner (2012): wide-column storage and globally consistent databases.
Dynamo (Amazon, 2007): highly available key-value storage. It uses consistent hashing (a way to spread keys over servers so few keys move when servers change) and quorums (a vote among replicas).
Kafka (LinkedIn, 2011) and Jay Kreps' essay "The Log": log-based messaging.
Scaling Memcache at Facebook (2013) and TAO (2013): caching and the social graph.
Dapper (Google, 2010): distributed tracing.
The Tail at Scale (2013): why the slowest requests matter.
Borg (2015): cluster management, and the roots of Kubernetes.
Raft (2014): an understandable consensus algorithm. Consensus is how several machines agree on one value even if some fail.
Zanzibar (2019): Google's global system for authorisation (deciding who may do what).

Books to go deeper:

Designing Data-Intensive Applications by Martin Kleppmann
System Design Interview Volumes 1 and 2 by Alex Xu (and Sahn Lam)
Site Reliability Engineering and The Site Reliability Workbook by Google
Release It! by Michael Nygard
Building Microservices by Sam Newman
Database Internals by Alex Petrov

Courses:

MIT 6.5840 Distributed Systems (lecture notes and papers)
CMU 15-445 Database Systems (free lectures online)

Build a learning habit

Read one engineering post or postmortem each week, and write a 5-line summary using the template above.
Map each post to the concepts in this series. For example: "This is really about hot keys (one key with far more traffic than the rest) and sharding."
Redesign it yourself: "If I had Discord's problem with my team, what would I do?"
Follow up: check whether the company later changed its approach. The follow-up posts are often the most educational.
Share what you learn. Writing about it (as I've done with this series) is one of the best ways to understand it deeply.

Trade-offs

Learning from big companies: battle-tested ideas and real numbers, but solutions tuned to a scale you may never need. Always translate to your context.
Reading widely vs deeply: many posts give broad awareness, while re-reading a few classics (DDIA, the Dynamo paper, the SRE book) gives deep understanding. Do both.
Following trends vs basic ideas: tools change every few years, but the fundamentals (caching, replication, partitioning, consistency, idempotency, failure handling) stay relevant for decades.

In the Real World

Every idea in this series came from somewhere real:

Consistent hashing started as an MIT research paper and helped start Akamai, a CDN company (post 26).
Kafka began at LinkedIn as a way to move data between its many systems in an orderly way (post 36).
Circuit breakers and bulkheads became mainstream through Michael Nygard's book and Netflix's Hystrix (post 42).
Error budgets came from Google SRE (post 47).
Blameless postmortems spread from Etsy and Google (post 48).
Kubernetes grew from Google's Borg (post 56).
Hybrid fan-out was explained publicly by Twitter's engineers (post 64).

Engineers sharing their experiences openly is one of the best things about our industry.

Interview Questions

Q1Tell me about an engineering blog post or postmortem you learned from.

Pick one and walk through problem, constraints, options, decision and result. For example, Discord moved from Cassandra to ScyllaDB. The problem was slow p99 latency (the slowest 1% of requests) and hard day-to-day operation with trillions of messages. They migrated carefully and checked the results. The lesson is that the migration is most of the work.

Q2Why be sceptical when reading big-company engineering posts?

They are written after success and often leave out the messy parts. They fit that company's scale, team and budget, which may be different from mine. They may also be out of date. Many companies later reversed the approach they wrote about.

Q3What is the most common trigger of major outages?

A change: a deploy, a configuration update, a command or an automated action. That is why canaries, feature flags, staged rollouts and fast rollbacks (going back to the old version) prevent more outages than any other single practice.

Q4What does Segment's “Goodbye Microservices” teach?

Splitting a system into too many small services creates extra operational work, repeated code and slow development. Segment merged back into one service, and work became faster and more reliable. Architecture should follow team size and workload, not fashion.

Q5How would you keep learning system design after a course or interview?

Follow a few engineering blogs. Read public postmortems regularly. Read the classic papers behind the tools you use (Dynamo, Bigtable, Kafka, Spanner). Best of all, build and run something small, then write down what broke.

Key Takeaways

Real systems are shaped by constraints, history and trade-offs. Engineering blogs, talks, papers and postmortems show how.
Read each post with a template: problem → constraints → options → decision → result, then translate it to your own context.
The recurring lessons are:
- start simple,
- migrations are the real work,
- architectures swing back and forth,
- cache carefully,
- most outages start with a change,
- design for failure,
- make retries safe,
- invest in observability,
- security failures are usually basic,
- culture matters.
Keep learning with engineering blogs, curated GitHub lists, classic papers, books and courses.
Build a weekly habit: read, summarise, map the post to fundamentals, redesign, and share.

Keep learning

Related series