Design a News Feed

Step 1: Requirements

Functional:

1. Users can publish posts (text, images, video).
2. Users can follow other users.
3. Users see a home feed of posts from accounts they follow, paginated with infinite scroll.
4. The feed order is reverse-chronological, or ranked by relevance.
5. Show like and comment counts and whether I liked each post.

Out of scope: writing comments, search, ads, stories, direct messages (post 63), and the details of recommendation models.

Non-functional:

• Fast reads: the feed loads in < 500 ms (p99), because this is the app's main screen. p99 means 99 out of 100 requests are at least this fast. Only the slowest 1 in 100 may take longer.
• Freshness: new posts appear in followers' feeds within seconds to a minute.
• Highly available: a slightly stale feed is much better than an error (post 27, choose availability).
• Read-heavy: far more feed views than posts.
• Scalable to hundreds of millions of users, including celebrity accounts.
• Eventual consistency is acceptable for feeds and counts (post 28). Eventual consistency means different copies of the data may differ for a short time, but they all become equal soon.

Step 2: Estimation

Assumptions:

• 100M daily active users (DAU). DAU is the number of different people who use the app on one day. Each opens the feed 10 times per day.
• 10% of DAU post once per day, which is 10M posts per day.
• Average followers per poster: 200. A small number of accounts have 10M–100M followers.
• Each feed page shows 20 posts.

So this means:

• Reads dominate (about 100× the posts), so precomputing feeds is attractive.
• Fan-out on write is manageable for normal users (tens of thousands of cache writes per second), but celebrities break it (50M writes for one post).
• The feed cache (lists of IDs) is around a terabyte. That needs a sharded Redis cluster. Redis is a fast database that keeps data in memory. Sharded means the data is split across many machines. This is affordable if we only keep feeds for active users.
• Media (photos and video) must go through object storage + CDN (posts 14 and 17). Object storage keeps big files cheaply. A CDN (content delivery network) keeps copies of files close to users.

Step 3: API and data model

API:

POST /v1/posts                  { text, mediaIds[] }     → 201 { postId }POST /v1/users/{id}/follow                                → 204GET  /v1/feed?cursor=…&limit=20                           → 200     { "items": [ { post, author, likeCount, commentCount, likedByMe } … ],       "nextCursor": "…" }

Use cursor-based pagination (post 34). Pagination means sending a long list in small pages. A cursor is a marker that says "continue after this item". New posts arriving at the top must not cause duplicates or gaps while you scroll.

Data model:

users:        user_id, name, avatar_url, follower_count, is_celebrity (derived flag)posts:        post_id (Snowflake: time-ordered), author_id, text, media_refs, created_at              → stored in a sharded DB / wide-column store, sharded by post_id or author_idfollows:      (follower_id, followee_id, created_at)              → two access patterns: "who do I follow?" and "who follows X?"              → often stored twice (by follower and by followee) = denormalised (post 23)feed_cache:   user_id → sorted list of (post_id, score/timestamp)   [Redis sorted set / list]counters:     post_id → like_count, comment_count                    [Redis + async DB]likes:        (post_id, user_id) → exists                            [for "likedByMe"]

The social graph is the web of who follows whom. It can live in a sharded relational or wide-column database, or in a dedicated graph store (post 20). Facebook built TAO for exactly this.

Step 4: High-level design

• The write path is fast for the author. The post is saved, an event is published to Kafka (a system that stores a stream of events in order), and the "posted!" response returns immediately. Fan-out happens asynchronously, which means in the background, after the reply (posts 18 and 39).
• The read path is mostly cache reads: a list of IDs, then hydration (turning IDs into full objects, such as the post text and author name) from caches, in batches (many items in one request).

Step 5: Deep dives

Deep dive 1: Fan-out on write (push)

When a user posts, write the post ID into every follower's feed cache:

Asha posts (200 followers) → 200 × ZADD feed:{follower} <timestamp> <post_id>

• ✅ Very fast reads. A feed is just "read my list", and reads are ~100× more frequent than writes.
• ✅ Simple read path, and a predictable read latency.
• ❌ Celebrity problem: 50M followers means 50M writes. These take minutes and flood the system. A "hot key" is one piece of data that gets far more traffic than the rest. Write amplification means one action causes many writes.
• ❌ Wasted work: many followers are inactive and will never read those feed entries.
• ❌ Slower "publish to all followers" for big accounts.

Deep dive 2: Fan-out on read (pull)

When a user opens the feed, fetch recent posts from everyone they follow, then merge:

Me → follow 300 accounts → fetch latest posts per account (from per-author timelines) → merge → top 20

• ✅ Cheap writes, with no fan-out at all. It's always fresh, and no work is wasted on inactive users.
• ❌ Slow, expensive reads: fetching from hundreds of authors for each feed view, at 35,000 views per second, is a huge amount of work. The total time is set by the slowest fetch. This is called tail latency (post 8): when you wait for many calls, the slowest one decides the speed.

Deep dive 3: The hybrid, which is what real systems do

• Most authors are normal, so most feed content is precomputed (fast reads).
• Each user follows only a handful of celebrities, so pulling their recent posts (which are heavily cached, because everyone reads them) is cheap.
• Inactive users: skip fan-out to users who haven't opened the app for, say, 30 days, and rebuild their feed on demand (pull) when they return.

Deep dive 4: The feed cache

• Redis sorted sets keyed by user: feed:{user_id}. A sorted set is a Redis data type that holds unique items, each with a number called a score. Redis keeps the items ordered by score. Here the score is the timestamp (or a rank score), and the item (the "member") is the post_id.
• Keep only the latest N (for example, 500–1,000) entries per user. Trim the list on each insert with ZREMRANGEBYRANK, a Redis command that removes items by their position in the order. Older content is fetched from storage if someone scrolls very far.
• Store IDs, not full posts. It's small, and edits or deletions are reflected when hydrating.
• Shard by user_id across a Redis cluster (post 25). Each user's feed lives on one shard. Add replicas (extra copies) so the feed stays available if a machine fails (post 24).
• Cache miss or rebuild: if a user's feed cache is empty (new user, eviction or returning inactive user), build it by pulling, then store it.

Deep dive 5: Hydration: turning IDs into a page

For 20 post IDs, we need post content, author names and avatars, like and comment counts, and "did I like this?":

• Batch fetch everything (multi-get from caches), never one by one. The N+1 problem happens when you make 1 call for the list and then N more calls, one for each item. It is slow (posts 5 and 32).
• Layered caches: a post cache (hot posts are read by millions), a user or profile cache, and counters in Redis.
• "Liked by me": check a set of (post_id, user_id) likes, cached or in a fast key-value store.
• Parallel calls (all at the same time) with timeouts (a limit on how long to wait). If counters are slow, show the post without counts instead of failing (post 44).

Deep dive 6: Ranking

Many feeds are ranked. Ranking means sorting posts by how useful they are to you, not only by time:

• Ranking runs at read time on a limited set of candidates, which keeps it fast.
• Pagination with ranked feeds needs care. Store the ranked session (or a snapshot of the candidates) so page 2 matches page 1. Use an opaque cursor, which is a token that the client cannot read or change.
• Keep a chronological option where users want it.

Deep dive 7: Counters, likes and consistency

• Like counts are updated asynchronously. Redis collects the increments in batches and saves them to the database every so often. This is called write-behind (post 16). Counts may lag a little, which is acceptable (post 28).
• Very hot posts (viral content) are hot keys. Use sharded counters (split one counter into several keys, and add them up when you read) and local caching (post 16).
• Read-your-writes for my own actions: after I like a post, I must see my own change right away. So my view shows it as liked immediately, even if the global count updates a few seconds later (post 24). Two things help here. Optimistic UI means the app shows the result before the server confirms it. The app also reads my own like state from the source of truth.

Deep dive 8: Edits, deletes, privacy and unfollows

• Deleted posts: because feeds store IDs, hydration simply skips posts marked deleted. There's no need to remove the ID from millions of feed caches immediately (a cleanup job can do that later).
• Privacy changes and blocks: check visibility at read time during hydration (a post author blocked me, or an account turned private). Never rely only on what was fanned out earlier.
• Unfollow: stop future fan-out, and filter that author's posts at read time (optionally clean the cache asynchronously).
• New follow: optionally backfill a few recent posts from the new account into the follower's feed. Backfill means adding old items that were missed.

Deep dive 9: Reliability and performance

• Availability over consistency: if fan-out is delayed, feeds are slightly stale, which is fine. If the ranking service fails, fall back to chronological. If counters fail, show posts without counts (post 44).
• Queue-based fan-out absorbs bursts (for example, everyone posting during a big match). A queue holds work until a worker is free. Monitor fan-out lag (how far the workers are behind, called consumer lag in Kafka, post 36) as a key freshness measure.
• Multi-region: users read from their nearest region's caches. Posts and events replicate across regions asynchronously (post 24).
• Prefetching: the app can fetch the next page in the background and cache the first page locally for instant app opens.
• SLOs (service level objectives, which are goals you promise to meet): feed p99 latency, fan-out delay ("99% of posts in followers' feeds within 60 s"), and error rate (post 47).

Wrap-up

• Key decisions:
  • Hybrid fan-out: push for normal users, pull for celebrities, and skip inactive users.
  • Feed caches of post IDs in a sharded Redis cluster.
  • Async fan-out through Kafka.
  • Batched hydration from layered caches.
  • Read-time ranking over a candidate set.
  • Visibility checks at read time.
  • Cursor pagination.
  • Eventually consistent counters.
• Trade-offs: faster reads (push) vs cheaper writes (pull); freshness vs cost; ranking relevance vs predictability; eventual consistency for counts and feeds.
• Next steps: recommendation-driven content from accounts you don't follow, ads insertion, video-heavy feeds, and real-time "new posts" banners via SSE or WebSockets (post 33). Both are ways for the server to push updates to the browser.